If someone can exploit your SMS, it's possible they can use that to social engineer their way into a password resets with services. (I forgot may password but I still have my phone.) So I would say a bad second factor can be strictly worse than no second factor.