Withdraw the book from further sales? Absolutely and no doubt. Withdraw it from people who already purchased it without doing anything wrong? Absolutely not. The legal questions are complicated and IANAL, but morally they had no right to remove something that was rightfully purchased and in use from people who had done no wrong.

letting people keep illegal content was out of the question here.

Hardly. The users did nothing wrong. Amazon (or arguably the publishers) made the mistake, the purchasers did nothing wrong. Any penalties (if proper) to be born should have been born by and only by Amazon.

Really? Because I've always thought that deleting pirated content would be a great punishment for piracy. If I've stolen something, I have no right to it. That's implicit in the idea of piracy. Deleting the stuff I pirated seems like a harmless punishment.

This is getting off on a tangent since this case is not about piracy (the customers did everything right and purchased the book, Amazon thought they were doing everything at the time).

But to go off on that tangent, no one should have the ability or right to delete anything or deliver any form of "punishment" without a judge being involved, and once it got to that point then simply deleting the files wouldn't be nearly enough.