Freedom of speech is not freedom from consequences. If people are free to speak, I am also free to speak out against them. If people support particular speakers, I am also free to speak out against those supporters, and to shun them.

If you don't believe me, maybe you'll believe a lawyer who does a lot of free-speech work: https://www.popehat.com/2013/09/10/speech-and-consequences/