This is what I do as well. Yubikey configured as a smartcard running gpg-agent with enable-ssh-support.