What prevents a compromised laptop from reading your private key in the clear from your smartcard? Even if you need a fingerprint read or PIN, I mean at some point you need to get the decrypted key read from the card so that it can be used.
Edit: For example, you can view what is currently running on the smartcard: http://i.imgur.com/Pgwz4la.png or view the filesystem on the smartcard: http://imgur.com/aLsV6pf