I do the same thing, plus I maintain a separate public/private key pair for every host where I need the pubkey auth.