Moral relevancy is kind of a cop out argument. We don't need literally everyone in the world to agree on something before we start taking action. Advocate for your own opinion, don't hide behind theoretical people across time and space.
Finally, I'm going to flip your argument a bit. You're the one who stated that there's no logical way to identify hate speech, so why don't you provide some examples? What are some things that you've seen identified as hate speech, where you disagree with that assessment?
How about advocating for the diminishing of people's civil liberties based on factors such as age, sex, sexual orientation, or religious views - that's difficult for you to spot as hate speech? Really?
We've passed the days of "cultural norms" and have entered the era of universal human rights. This is a recent transition, but it isn't hard to spot hate speech or hate crimes.