Require both the physical device and the tap code, mabye.

Maybe apply some pressure instead of tapping? Same effect, but with less sound, so you'd need a really good mic, or something crazy like pointing a laser at the device/hand, and if that's the kind of attacks you're facing, you don't go for a device that costs 30$ xD

Like one of these? onlykey.io