There is technically one minor one, which really isn't one - but you should be aware of.
Someone can take your authorized keys and add them to a box they control, and trick you into logging in.
However, this would trigger the "new host" warning SSH gives you, and you can minimize this by minimizing which hosts you allow your private keys to be used on.
And if someone is so actively trying to attack you they probably have more direct methods available.