[An imagined conversation between the Bitcoin novice Glaucon and Satoshi himself]
Glaucon: Look, I just bought my first bitcoins on an exchange! Can I just keep them here? Are they safe?
Satoshi: The short answer: no.
If you bought them on an exchange and you still have them there, they're somewhat safe over the short term, but I wouldn't trust them there for long.
Keep in mind that when you keep your bitcoins on an exchange site, you don't actually have possession over any bitcoins at all - you really just have an IOU for the amount of bitcoins you bought.
If the exchange gets hacked or goes out of business, all your money is gone, and the exchange can't get it back. And if you're dealing with a foreign exchange and the founders run away with your money, there is little to no recourse.
Bottom line: if the bitcoins you purchased are still on an exchange, you are bearing a decent amount of risk that you won't be able to get them back.
Glaucon: And the same goes for online wallets?
Satoshi: Precisely. Online wallets are very risky - quite a few wallet sites have been hacked already, so I wouldn't store any sizable amount of money in them (probably no more than the cash you keep on your work desk).
Glaucon: How about Blockchain.info or Coinbase? I heard they're a bit different.
Satoshi: That's true.
If you use Blockchain.info, they never have access to your funds - instead, you safeguard a passphrase that has the power to unlock your wallet and release the funds. Right now, Blockchain.info is considered the safest web wallet.
Meanwhile, Coinbase is safer than other online wallets because it (purportedly) only keeps 3-5% of it's funds online, with the remaining funds stored offline. Theoretically, if Coinbase were to get hacked, they'd lose a maximium of 5% of their funds. This, in addition to the fact that they're based in the United States and have a strong financial backing, makes Coinbase the other good choice for a web wallet.
However, if you have a decent percentage of your personal savings in Bitcoin and/or you want a higher level of security and control, I'd seek a more rock solid solution.
Glaucon: Hm, what would you consider rock solid?
Satoshi: A high security solution requires some flavor of offline storage, also known as "cold storage."
However, it can be quite difficult to get cold storage right, and it's important to do things by the book because we're dealing with our personal wealth.
Glaucon: Ah, I see your point, but I'm having some trouble grasping how bitcoin storage works. Can you shed some light on this?
Satoshi: One of the most beautiful aspects of Bitcoin is the remarkable ease with which you can store and transmit money.
In fact, the only thing you need to do in order to spend your bitcoin (and prevent others from spending it) is to remember and protect a secret number: your private key.
Think of this key as analogous to the PIN number of a bank account, with the caveat that you don't have access to any customer support or account recovery services.
With Bitcoin, if someone discovers your key, they have the full power to move your money wherever they so choose; if you lose your key, your money is gone.
Further, if your key is not truly random, then someone could very easily guess it and all your funds will be stolen.
So evidently... the feat of remembering and protecting your private key is much easier said than done.
Don't be scared away though! If you follow all the best practices, if you generate a truly random key and keep it safe, your funds will remain completely secure.
Glaucon: Whoa, you've got my attention. Now how do I get started?
Satoshi: That, Glaucon, is for another discussion. I will assist you with your Bitcoin storage needs in due time.