I don't think this is as true as you claim. While the word toxic isn't used, black culture and to a lesser extent gay culture are often described as toxic or harmful, either to those groups or to society more generally.
This is even more true if you look at, e.g. transness, where an article about how it's contagious was in the NYT this week. And, IME, there's usually agreement that men aren't solely to blame for toxic masculinity, while allusions to these other toxic cultures are usually used to specifically avoid addressing issues that affect those groups.