Yubico already make a Security Key that isn't also a PGP key store, a TOTP authenticator, bagel toaster and whatever else for about $20. And there are cheaper vendors if price is the main concern.
The argument seems pretty straightforward: if you don’t trust SMS, you need to disable all backup authentication. If you’ve disabled backup, you surely don’t your physical device to be a single point of failure?
It's the backup option, it's the reserve, it didn't have to be the best possible thing, just enough to get you back into the game after you lose the main token somehow.