If the question is framed, Should I stop someone trying to stop someone from exercising their freedom of speech? Definitely. This is the paradox of tolerance. https://en.wikipedia.org/wiki/Paradox_of_tolerance
Depending on whether the other group tries to undermine you.
We surely like freedom of speech but still we want to silence, for example, anti-vaxxer.