The Advogato trust metric sorts users into four kinds, "seed" (the built-in trusted users), "good" (those the algorithm assesses as both behaving well and certifying well), "confused" (those the algorithm asseses as behaving well but certifying erratically), and "bad" (assessed as badly behaved).
I think the problem you describe is in essence the need for an automatic trust inference that recognises that worthwhile participants often certify badly and knows how to "prune" the set of tolerated participants down to those whose voting activity is high quality.
Advogato hasn't really been successful overall (Steven Rainwater suspended new account creation a year ago), but there are many successes in its history (it has been run with very little intervention for over 10 years, and generally succeeded in separating spammers from free-software types for most of that period) and its experience is probably worth studying.