I'm inclined to agree with that as well. It is awfully convenient to just store your info and then buy with a single click.
To be honest, I can't really decide what I'd prefer. Be lazy or more secure.
At the very least they shouldn't store all info, as I know many places do.
I remember Play.com not even giving an option to not save card info, let alone delete it after a purchase. I hate that, a lot.