No! Aspartame is known to cause serious issues for decades now, even before it was finally approved by the FDA.
https://www.health.harvard.edu/heart-health/sugar-substitute...
There's a good list here: https://www.efsa.europa.eu/sites/default/files/event/documen...
Given the glut of well-documented harmful effects of sugar, I'll take the "kinda sorta maybe" of aspartame any day, because I'm not about to give up every pleasure in life just because they all seem to kill me.
Thing is, if by 'they' you mean processed foods, you are basically right.
A little bit of sugar that you add to a coffee or whatever won't kill you. The problem is that just about all the processed food available has been spiked with sugar.