I use one key everywhere, but the private key lives on two places only: a usb drive on my keychain and on a backup floppy. When I sit down at a system, I load the key into the agent and then unmount the usb drive. It seems that the only way the private key could be compromised is if someone breaks into the agent when I'm on the machine or if someone spoofs the connection to the agent (which, if I understand correctly, won't give them the key, only the ability to use the key while the agent is running).

If the usb drive is lost/stolen, there should be no information that reveals which systems it works on. The key also has a very long passphrase, since I only type it once per day, so that should at least make brute force attacks against the key difficult.

This seems like a pretty secure solution to me. What other weaknesses can you think of?