This is extremely risky for the integrity of the remote copy. If the key is compromised (USB stick lost or acquired by a bad faith actor) then the remote repository is untrustable.
I suppose this is no different to normal keyloss, and some people maintain their keys on removable devices and are exposed to this loss, if the device does not have additional protections.
If it's not a bare (private) key, I suppose then it comes down to the ssh-agent chain over that key, and the strength of your wrapper protection.