If you got passwords, users either pick easy ones or write them down next to the device that needs them. If you require physical "key" items, user leaves the key next to the computer that needs it.
When designing a security system you need to acknowledge this limitation, and design the system with it in mind. Running between "something you know" (passwords) to to "something you know and has" (Two-factor authentication like password and phone), and now back to only "something you have" (an USB key) won't solve the problem.