The answer is that this is, in fact, a slippery slope, because though TDS seem to be something that could be reasonably called actual Nazis (I honestly haven't investigated them to determine the validity of the accusation), we've also got an enormous cottage industry with a thriving amateur sideline of political activists calling anyone they disagree with Nazis. And it's entirely possible for the "X is a Nazi" narrative to get off the ground and self-perpetuate without any actual truth to it, with no one involved actually examining the claim. A lie can run around the world, &c.

If you agree to censor any category of thing, then you'll immediately have a large group of unscrupulous people attempting to cast all their enemies as that thing. The only general countermeasure is to not censor anything.