Privacy concerns aside, this is one of the primary reasons why I try not to give my phone number to websites I sign up for. I can't trust them not to treat it like an authentication mechanism. OP didn't want to use his phone number as authentication. This was a setting somewhere that got enabled by default, even though for the most part, nobody should ever have it enabled.
Why does this setting exist?
It really feels like a juvenile security mistake to me, and I don't understand the reasoning behind Twitter's security team being OK with it. To me, this seems like a mistake on the same level as using security questions or mandating password expiration. Maybe there's some justification I'm missing, but right now it's difficult for me to imagine what it would be.