I use my yubikey as GPG key for mail and for pass(). I really like having a backup.
If you use 'pass' for storing passwords, then (A) will make your password store unavailable.
And (B) is a lot of places you have to update. In fact I aim to make it impossible to connect and update my servers key without having the existing key.
-- In short, since sub-key infrastructure really works for signatures, it seems easiest to just backup your subkeys.
openssh supports a CA model, so .. you can set it up so that the servers trust the CA and the CA blesses the new key. Obviously, it just shifts the "but what if we lose the .." question from the user key to the CA key, but in many cases it makes a difference.
Anything that can work with GPG and GPGAgent should work with a Yubikey