We have always made a distinction between things that are not the fault of the individual (skin color, gender, ethnicity) and things that are (speech and actions). Which seems perfectly reasonable to me.
Free speech is a good idea for game-theoretic reasons. That's it. Free speech lets people fight it out with only the occasional riot and attempted overthrow of the republic. That's better than the alternative.