This can go both ways. A majority view may be held complacently without being challenged much or often. A minority view encounters constant resistance and reminders that it is not held by many, and may potentially be more robust because of that. Either way, a belief ought to be based on facts and sound reasoning rather than group pressure, and even the most strongly held position ought to be held provisionally and subject to revision in the light of human fallibility.

If the first few people to see something downvotes you into oblivion before the majority arriving later get a chance to see your content, that's silencing, not merely not being listened to. This is doubly true when those first few people are likely to be regulars that refresh pages often. This means that a few people that are ideologically on the same page can drive the ideological view of an entire forum without actually being a moderator.