The flagging system is almost there from that point of view, but it's still pretty hit and miss... I think if the flagging system was tweaked to give more weight to the flaggings of people on the leader board, the system would work extremely well.
Of course, this may be by choice - after all pg may not want to create an oligarchic moderator class on news.yc... up to him.
I'd see an oligarchy as more of a site like Reddit or News.YC, but rather than every person having a vote, a set group of people are able to. Like if only the leaderboard could perform certain tasks.
The problem to circumvent then would be if the leaderboard became overtly political, but I think that a) that isn't going to happen on a site like this, and b) it should be easy to make the system balanced if we have Dictator Paul Graham around to make sure that the people in such a system don't turn into absolute jackasses.
The problem lies when normal people stop thinking that they can submit and get content to the top, the submissions tend to become rote, and you lose community interest.
(As an aside some of the power users on Digg started taking bribes and were banned en-masse IIRC, but I don't see that happening here.)
It won't happen, but I think it would be an interesting experiment.