Normally they're used for OTPs or Chrome's FIDO U2F auth scheme, but you can also put them in smartcard mode. If you do this, you can use it as an OpenPGP key, which is also compatible with SSH:
https://www.esev.com/blog/post/2015-01-pgp-ssh-key-on-yubike... https://www.yubico.com/2012/12/yubikey-neo-openpgp/
Just about the only place that won't work is on a phone. Technically it has an independent NFC interface which could probably be made to work with some extra development, but I haven't looked into it.
That way a brute force attack would essentially need to gain access to the shared OTP secret in order to be successful (assuming ~few/normal user population. With a 6-digit otp, I suppose the value of otp drops if an attacker can try to log in as ~1 billion users in parallel -- but it's not really a scenario most of us have to worry about).
Another option is to use ssh certificates. As long as you are able to make yourself new user certificates, you can log in. One bonus with certificates is that they can be set to expire, unlike keys.
If I lose the phone, there's a pretty good password on that key that should give me ample time to remove access rights for it.
But at that point, we're probably also worried about the odds of getting run over by a bus and having amnesia that prevents you from using a password protected key, chance of getting struck by lightning, etc. ;)
Yes actually. Because it has a copy of my password manager file, which has all the passwords to the various social sites I could use to alert friends that I need a ride home. ;)
Then if you need to get in, download the key, type in the password, login as that user, and the su up to root by entering the password only you know.
As an additional layer of security you could encrypt the key using AES-256 (e.g. 7Zip archive, Microsoft Office's Word .docx (NOT .doc) format (just drag drop the file into Word, "Encrypt with password") now AES-256 encrypted, others).
Now you have four layers of protection:
- su password.
- Key password.
- AES-256 encryption password.
- "Cloud" storage credentials.
(A stepstone is a well-protected server where you can log in with user/pass and use your keys to login to other machines from there.)
Also note that it's not really an extra step, you put in a ProxyCommand in your .ssh/config file.
Don't note what machine or where the key has access to either. I know purists will take issue with this but it's a solution to a problem as you have detailed.
Obviously only works from "borrowed machine from friend work etc" unless you can get the key from a machine to "wife's phone".
You still need a passphrase to decrypt the private key. If you bump up the number of trials, you can make decryption even slower. It's going to be harder for someone to crack that than a normal brute force against a password.
Now, if you put that key behind a decently secure storage service that you use TFA to access, that's actually way more secure than just using a password.
Given that, the question ought to be whether you upload the private key to the cloud, or try to manage the private key yourself. Password based authentication shouldn't be one of the options you are considering.
They will have to do a session/key exchange hand shake each time, but that shouldn't be more painful than the extra work needed to crack a private key with tons of trials.