There is probably a good reason for this that I don't know, but why haven't physical keys replaced passwords yet? Want to manage your IoT device? Just slot in a USB key, do your thing, and unslot it.
You don't really have to sweat the security of the key laying around because its mere existence requires an attacker perform physical penetration which is a different attack profile from someone just cruising Shodan.
Granted, a USB key doesn't make sense for every device, but e.g. it seems like it would make a lot of sense for my router.