This site needs some kind of meta-moderation system, or some other way to moderate the moderation.

What you describe seems to be happening more and more often these days, and I think it's harmful. Comments will receive many upvotes, and then suddenly start getting many downvotes. Or it'll happen the other way around.

When that kind of fluctuation is happening, I think the only appropriate thing to do, especially when downvotes or downvoted comments are involved, is to show the comment normally, instead of obscuring it with harder to read grey text. Clearly there's some agreement, and there's some disagreement, so the only sane thing to do is to show the comment so readers can make their own analyses of it.