Think of it like a doctor selling you a medicine, then calling you up and saying, "Oops, I wasn't authorized to give you those pills. I need them back, and I'll refund you entirely."
There's a difference between removing an e-book and refunding it.
The doctor wouldn't break into your house and take the pills out of the medicine cabinet.
Perhaps I don't have the same feelings regarding owning a copy of somebody else's work. I don't feel ownership towards ebooks. I feel ownership towards the money I spend, but if I get that back I wouldn't be mad at what Amazon did. That's a minority opinion, and I think it would be worth debating somewhere that's not here.
This of course implies that the doctor would be within his rights if he caused no disorder. "I wouldn't trust a doctor to...". You didn't trust Amazon to either -- they just went in and did it, at someone's request.
Hmm.. why does this prescription bottle have an antenna on it...