You can use your yubikey to store your subkeys (still generated on the yubikey itself) while your master key is stored somewhere else. If you lose your yubikey you can revoke the subkeys and generate new ones signed by your existing master key.
If you use 'pass' for storing passwords, then (A) will make your password store unavailable.
And (B) is a lot of places you have to update. In fact I aim to make it impossible to connect and update my servers key without having the existing key.
-- In short, since sub-key infrastructure really works for signatures, it seems easiest to just backup your subkeys.
openssh supports a CA model, so .. you can set it up so that the servers trust the CA and the CA blesses the new key. Obviously, it just shifts the "but what if we lose the .." question from the user key to the CA key, but in many cases it makes a difference.