SMS as a
second factor is not a net reduction in security. The ability to hijack a phone via number porting or similar could give access to SMS messages, but by definition a
second factor should never
reduce your security.
What does reduces security is the use of SMS as a password reset mechanism, or any similar method that uses SMS as the only factor for authentication. Don't do that.