Sure - "The loud fanatics" who have no following certainly drop into irrelevance. I think if what you say were completely true we would not have had any movement forward on positions that were considered radical just 10 - 20 years ago - like same-sex marriage. It is only because certain people with large followings - Andrew Sullivan and others - began making a vocal argument for gay marriage, which wasn't even considered a mainstream position until very recently, that it is now the law of the land in so many states.
Radicals have to be able to articulate their position in a way that is compelling and reasonable to a significant number of people in order for there to be a shift. But I think political progress is largely explained by this phenomenon.