Because neither are numerous enough so as to constitute a legitimate threat.

If we started seeing local governments having openly KKK or neo-Nazi majorities, the freedom of speech balance would substantially alter.

First amendment rights in the US are typically circumscribed only via a requirement to show actual, existential harm.