If you want to get theoretical, look up paradox of tolerance. If you want a practical example, look up 'how to deal with nazis at your bar' [1]. Basically you get stuck with the assholes as the good people get silenced or leave. It's a pattern as old as time itself.
https://www.reddit.com/r/TalesFromYourServer/comments/hsiisw...