> I'm sure the recovery process is not as easy.

A lot of services offer one-time backup codes and connecting multiple 2FA devices. Making Yubikey a single point of failure is certainly a bad idea.