Stealing or guessing a password is infinitely easier than compromising a Yubikey or otherwise hardware-based authentication factor. This is a demonstrable fact and any argument to the contrary is simply pearl clutching. I challenge you to find even a single incident resulting from a compromise of this 2nd factor, until you do there's nothing to discuss. And because you don't find it, it will ultimately demonstrate passwords are not even necessary when accounts are protected in this way.