That’s not scalable. But maybe a 1/N probability of being required to do so, for reasonable N.

(Edit: something like slashdots categories could work but you’d need some incentive to be truthful rather than trust them to separate a “troll” from a “disagree”.)

Plus, if anything, I’d say it’s more common for there to be a bias against downvoting relative to upvoting.