I empathize with your anxiety about losing a physical security device. I feel the same anxiety about losing access to my password manager. What helps me manage my anxiety is having a solid backup framework. The same applies to physical security devices -- having additional backup passkey devices help make me feel more comfortable that I'm not going to lock myself out of anything important.