I configured it so I can only sudo with my password. So even if someone gets hold of my private key, they'd have to guess my password for sudo access. Also, as someone pointed out you can restrict sudo commands, which is the sensible thing to do.
SSH to your account, add an alias to sudo in .bashrc (or equivalent for your shell) that records the password in a hidden file before passing it to the real sudo.
For example, the attacker could inject a LD_PRELOAD library that redirects execve() calls from /usr/bin/sudo to a fake version of sudo.
Or, if the server is running Debian, you can set environment variables the .pam_environment file:
The .pam_environment one is interesting, thanks for the link.
By the way, this is why Microsoft invented the UAC prompt: rather than a password, which can be intercepted, it generates a window which (presumably) cannot, thanks to User Interface Privilege Isolation.