Don't forget that makes you more vulnerable to malware on the client, which now has easy access to your keys, and your known_hosts file (which gives the attacker a convenient list of servers to attempt to log into using those keys.)
Yes, this is why having a passphrase on your key is recommended. The idea is not to use keys instead of passwords, but in addition to: the password protects the private key locally, which is used to authenticate to the server. However that seems to be uncommon practice.