What happens when/if your private key is compromised? All accesses will be compromised All accesses need to be rekeyed
Some factor that's unique to the service would help to minimize this, something of an unique extension to the public/private key, something like two keys for the service, the general one and the other one (this is something other than your password, so the server would still need no password)
Tooling is key . Not only being able to manage all those public/private keys but also backing them up and make sure that they aren't lost. Or perhaps using email or other channel to reset them
Keybase could probably be an excellent tool to manage such key.
This is still a tad raw, and somehow user experience isn't factored in. We have a very powerful computer with fingerprint recognition in our pockets, why do we still need to type in passwords? (think WhatsApp web login)