He persuasively advocates that the line shouldn't be drawn based on the acceptability or reasonability of the opinion, but on whether the behavior is mere communication (i.e. leaflets, speeches, opinions, blogs, TV ads) or actual action. In particular, when the causal influence of the alleged wrong doing propagates through the minds of voting citizens, this is not something that ought to be retaliated against (by the government or by individuals).
Incidentally, Mills strongly argues that speech should be protected even under the assumption that it is wrong and that its suppression would not have slippery slope effects. If you can't see why what would make sense, I encourage you to read him.
For more on arguments like Mills, as well as some modern push-back grounded in the principle of offense, see the Stanford Encyclopedia of Philosophy.