If someone has remote access to your computer then either way you are screwed. Using password or key file won't change anything.

If your laptop get stolen, then just remove the corresponding line in your authorized_keys file.