Why would you want to store yubikeys securely, as opposed to recovery codes which you can print out in multiple copies? Store it in multiple semi-secure places. Unless you're running infra for an international corp, government, bank, or are likely to be physically targeted for some reason, you can likely store it in a folder on a shelf.
(And if you actually need to worry about things like that, then you've got (or should have) people who think of things like that for you)