make sure one's ssh-agent forwarding is off when connecting to strange hosts :)
Host * ForwardAgent yes
The scenario you talk about is certainly possible, especially since OpenSSH does not require a "tap" for each authorization once the agent is unlocked. (Side note: this is one thing hardware u2f keys absolutely got correct).