Merely expressing an idea, even if it involves negativity or dislike toward a certain group of people, is not in itself harmful. Now, harm may arise if somebody in turn acts after being exposed to such material. But the responsibility for that harmful act should lay solely with the individual or individuals who perpetrated it, not with people who merely expressed ideas that may have been observed by the perpetrator.
It is harmful in and of itself. I don't know if you've ever had people urging others to hurt you but it's immensely threatening, very scary. And for good reason - there's the knowledge that there's easy respect up for grabs in that hateful little group, something that encourages lonely bitter people to go do the things. The speech creates an incentive for the action.
It's a difference of degree, not of type, between putting out a hit on someone and hate speech. It's just the payment that's slightly different.
If an act does occur, it's because one or more people decided to made that act happen.
The words in a book, or the sounds in a recording, or a JPEG image on Facebook have no inherent power to control people. Nor do the person or persons who created such works.
If somebody performs a specific act after viewing such material, it is that person who is responsible for any harm that may occur.
I asked someone to pass me the salt, and they passed me the salt. I can only assume you believe there was no causal connection between those two events.
The truth is that everything people deliberately do is caused by what they think, and what they think is very strongly affected by the symbolic communication they have with others. That is, after all, the whole point of language: to alter someone else's mind-state.