For a little while, I used a YubiKey NEO-n as an OpenPGP smartcard, with gpg-agent running as an ssh-agent (so that my SSH key was only present on my smartcard).
While great in theory, the authentication time made it totally impractical to connect to multiple servers at once.
I know this doesn't invalidate the general idea of using gpg-agent as ssh-agent -- just an anecdote.