The SSH agent shipped with OpenSSH already includes PKCS#11 support (check out the `-s` flag). I'm using that daily to work with TPM-backed keys.