Similarly we should favor even "bad" people being allowed to speak because norms change, often very quickly. 15 years ago it was entirely within the mainstream politics to not publicly support gay marriage. Now if you're against it you're labelled a homophobe. What views do you have today that might be decidedly distasteful in another decade? Do you know? Do any of us?
Suggesting that because you're expressing what are now mainstream views means you're reasonable and thus safe is to -- at least to me -- completely misunderstand the ideas behind equality under the law or the real goal of freedom of expression.