By design, talking to to the ssh-agent should not give you the decrypted key. However, in reality, anybody that could write to the ssh-agent socket could also ptrace() the agent and find the key in its memory. That would of course require becoming the user you are running ssh-agent as, but that's typically not very difficult.

What other weaknesses can you think of?

A key logger.

Basically, if you don't trust the system you decrypt your key on, all bets are off.