If someone was to obtain a copy of your vault, decrypting it would be trivial with a weak or compromised master password in that case.
CTAP supports an extension called hmac-secret that would allow you encrypt your vault, which would mitigate this issue (While introducing others potentially -- for instance, hmac-secret does not require user verification so anyone with your yubikey could decrypt it). Of course there are other mechanisms to encrypt a vault other than a key derived from a password that you can use with a Yubikey, like PGP, but I don't know of any commercial password manager that does it that way.
The thing I worry about is that the security of the passphrase is only as secure as the mechanism guarding it. If it's written on a piece of paper, then how is the paper secured? It could be put in a vault, but then the vault itself is a conspicuous target for thieves. Hiding the paper somewhere is probably pretty reasonable, but if it's too well hidden, it could get forgotten or accidentally thrown out over time.
I use a YubiKey as well as an ultimate backup, and it has a PIN code mechanism that will lock after a few incorrect attempts, so that is a reasonable tradeoff for security and usability I feel.
I wonder if the best solution is to have a distributed copy of a recovery passphrase to friends and family. Then separately have a distributed copy of the vault itself (in my case, it's GPG-protected Password Store). This must have been an area of study already, I need to do some research!