But then it would have to either keep the passphrase in memory (just as bad) or ask for the passphrase each time you reconnect later (defeats the purpose of ssh-agent).

Edit: Nevermind, you're talking about ssh, not ssh-agent..

You're right, I forgot that it would switch to asymmetric crypto after the handshake. My bad.

Still, with root it would be trivial to attach a debugger to the daemon, et cetera.