If you have Verizon, they have a number lock feature (https://www.verizon.com/support/port-out-faqs/#) that is designed to protect against this very thing. I have it enabled and feel much better about using SMS 2FA/recovery.