Then you should also be worried about bugs that let you log into an SSH session without providing your SSH certificate, passkey or whatever. Authentication bypass can happen with pretty much any buggy authentication method. None of this is inherently a problem of passwords or basic auth.