If someone gets hold if your smart card, they can enter a wrong pass phrase to remove your private key, right? So, you have to have multiple of these smart cards (that all share the same private key), some in your wallet, some in your house and some in a bank or vault somewhere. Otherwise, you can lose access to your private keys forever.
Am I right, or I'm missing something?