I have no right to complete silence in my apartment block, but I do have a right to not have to listen to my neighbor's loud music at 3am in the morning.
I have no right to never breathe in second hand smoke, but I do have a right to not have to breath it in inside small enclosed rooms.
Our society and the law makes these kinds of trade-offs all the time.
Also, the issue isn't people having to read the speech, it is the speech negatively impacting people (sometimes severely if you are an Asian person that just got knifed on the street) in indirect ways. There is this false belief that speech can't be mortally harmful unless it's a direct call to violence, which I hope we can dispense with since the evidence doesn't line up with that narrative.