If we have validated our users mobile numbers via a TOTP code, informed them that they are sending SMS as their number, then how is it different to an application like Signal or iMessage linking your messages to your mobile number? It's not.
And that's why it's allowed with certain SMS gateways. The SMS gateways are doing the due diligence to ensure you are sending spoofed messages appropriately and only as the user who actually owns the number.
Edit: The spammers are most likely using gateways who aren't doing these checks. The solution is to fine or shut these gateways down.