SIM swapping is one thing, but the actual service (X in this case) allowing access to the account via access to the phone number, even without SMS 2FA enabled, is the real problem.
For example, I have a foolproof way of preventing sim swap attacks: require 256bits of entropy and never allow a password reset, like in crypto. Lose your password? Account is gone forever.
This is more secure but less user friendly. Except for large accounts, I don’t know that anyone even particularly cares if their Twitter gets hacked. You could pretty easily make the argument that preventing sim swap attacks is an optimization for high profile users at the expense of everyone else.