Hate speech is always an attack.
Whether it's overt (Westboro, the N-word, Swastikas, homophobic slurs, etc.) or more subtle ("the 14 words", "88", etc). Supporting the "right" of someone to use hate speech is supporting their right to attack (and thus violate the rights of) others.
Whereas, by preventing those attacks, nothing of value is lost, and real harm is prevented.