What damage will someone do? I guess you risk an occasional transaction you don't know about. Someone can hardly hit 100's of transactions quickly without flagging something. Also I'd be surprised if the banks don't already/shortly tie your location to the transaction as the swipe is increasing done from the phone (or they have your mobile banking app even if card based).
One thing I'm trying to change is get my partner to stop using a clone credit card they have of mine. It makes it hard for me to do my monthly scan of spending an know what what should/shouldn't be there.
Going forward I hope they get an e-ink screen on the card showing the last spend amount. I dislike when businesses swipe small transactions and you cant see the terminal amount. I generally ask for the receipt in these cases but having the most recent spend on the card would make things that little bit easier.