Which is better: to have someone hack into your machine and install a keylogger and wait for you to connect to your servers using SSH to get your passwords or someone hacking into your machine and stealing immediately the private keys!?
Why would you have private keys on the server? Those are passphrase-protected on your laptop/workstation.