One common method is to secure the hardware key with a PIN. The key will only do its work (e. g. sign things) if the correct PIN is entered. It will also destroy the secrets it keeps (or even itself) after a given number of failed attempts.

If this is not feasible for some reason, you need to change or revoke your secrets as soon as you know the key is stolen.