I sure wouldn't. Many of the classic, non-nutritive sweeteners like sucralose have been tested more thoroughly than almost any substance we ingest. Even at heroic doses tens of thousands of times higher than what would be used normally, no significant health effects have been noted.
In contrast, we have sucrose: a caloric carbohydrate that rapidly and reliably raises blood sugar, and due to its fructose content, has been linked to NAFLD. If nothing else, it's 'empty' caloric value, high palatability, popularity in beverages, and low cost (especially as the almost functionally identical HFCS) has without a doubt been a major contributor to the obesity epidemic.
So sure, removing all added sweeteners ('natural' or otherwise) would be better, but a NNS is way, way better than sugar.