If the remote host is trusted, you just forward the gpg-agent over ssh to your remote host.
It's generally preferable to use a `-sk` key type, though, by which the remote server can essentially enforce that you're using a smartcard and not a normal keypair backed by a file.