Anything else could be lost. I can always get new SIM card for this number. I don't need to backup it and I can't accidentally delete it. That's the biggest reason for me to link phone number everywhere. I'd hate to lose access to my GitHub account.
I don't see how I could simultaneously lose my three hardware keys (laptop, phone and Yubikey) and backup codes.
It's also not very hard for scammers to get a SIM card for your number, unless you're using a carrier that specializes in not allowing SIM swapping attacks.
I dislike SMS 2FA and services that use my phone number as a stable identifier, however SIM swapping is not really a thing in most countries.
That's why 2FA is called 2FA. It should require two factors. SMS + Password, for example. So scammers would need to steal both password and perform SIM swap, which hopefully is a bit harder.