Yes, that's how it is, and I find it completely reasonable that people will hate you for wanting to reimplement segregation or slavery.

I don't know about US law, but the German constitution defines a clear hierarchy of basic rights, starting with:

1. dignity of mankind

2. right to live and and freedom from physical harm

3. equality of every person before the law

4. freedom of religion

5. free speech

So there are a lot of things that our law considers more important than free speech. This setup is a result of the German experiences with a certain kind of demagogues. I will not argue that this is the perfect law system, but I stand by it nonetheless.