I think the issue here isn't "freedom of speech", its that people who claim to want "freedom for speech" are either using it as a shield to say vile things to other people, or they feel that "freedom of speech" is the only thing one needs to guard against fascism.

The resulting difficulty is that the former is demonstrably true, and the former is demonstrably false.