Your bank imposes withdrawal limits. I imagine it is for this very reason. This would be a interesting idea for Bitcoin. On Ethereum this might already be possible with a smart contract.
There are also card lockout procedures (usually not destroying funds) and dispute procedures.