Passwords should always be kept as a valid authentication method.
The OP brings up a point about not having access to the private key... I agree with that. If I don't have access to the private key (like I currently do with ssh) then it's dead in the water to me.