Hardware tokens are a PITA. Sure everyone has a house key because they only have a house at a time. I have 3 bank accounts, a few brokerage accounts, some pension logins on top of the regular stuff. I'm not going to carry 15 hardware tokens with me.
Of course, that breaks the UX analogy of the house key.