Defamation has a singular victim. Hate speech has a class of victim. Why should we allow one and not the other? Why can't we just scale up our defamation laws to also apply to groups of people?
You could easily use such a justification to ban anti-war speech (something that has been done before), or ban speech against whatever you favorite political view is
Also we could just limit this to protected groups as they are currently defined for discrimination. That solves your problem with anti-war speech or generic political views.