Any time you reserve the right to upset people, there's a question of whether you're doing so because you're altruistic or because you're just cruel, or at least so insensitive that you prefer casual cruelty over bothering to be sensitive. Being protective of the right to upset people is an implicit judgment that sensitive people ought not participate in public discourse.
And maybe you're okay with that, because the hedge-against-consensus angle is so valuable, but I think the standard counter-argument is that the actual result of a free-speech approach is a yet-unhedged consensus determined by the insensitive jerks with the stomach for a public discourse privileging insensitive jerks.
This is orthogonal to the "this meme is so dangerous it must be suppressed" argument. One thing you can do if you want more acceptance to a pro-free-speech viewpoint is actively address the exclusion-of-sensitive-people angle, and try to decouple "having controversial ideas" from "being a jerk about it."