What needs to be removed is the password system entirely. I cannot believe that in the age of public-key cryptography we are still using passwords for serious security.
It's the old dilemma of machine-only protocols vs protocols that can be spoken by a human typing on a keyboard.
Say, can you tell me what the recovery process is if your Yubikey croaks while you're on a trip away from home?
Every option has drawbacks.
As for the yubikey croaking, easy: carry a backup.