> Passwords should always be kept as a valid authentication method

Why?

> If I don't have access to the private key (like I currently do with ssh) then it's dead in the water to me.

You don't have 'access' to the private key for any well designed HSM either. And there's nothing in the passkey spec that says you need to put the private key in at inaccessible place.