Thats exactly what Karl Popper says.
We should listen to them, aslong as society is strong enough to defend a tolerant society against intolerance.
The question is: When do we reach the point where the defence starts to break?
You could argue that that point is reached when terrorists storm the capitol.
I don't know if we reached that point already. What I do know: If we wait for too long, we reach a state of no return.