This is partly why I really liked the idea of Slashdot's meta-moderation. Although it could probably be gamed by enough determined people.

Moderators should be moderated too, because just like humanity in general, there's going to be a spread of skill levels among the mods.

I wonder if there's ever been a site that has "moderator karma", a score/scores by which a moderator's "performance" can be measured. History of that score too would be interesting to see, potentially showing if a mod has deteriorated in their judgement lately, stuff like that.

It feels like there could be more done to find useful data about moderation decisions.