I think reddit has the best system for ranking comments and threads of any I have seen. I haven't studied the source, but it seems to hinge on what I have taken to call 'piloting' new posts: Allow it a brief time in the top spot (possibly only for a random subset of users), and see how well it performs (upvote wise) compared with the other comments. And, importantly, the quality requirement for the post increases the higher in the total hierarchy it is: root-level post, top-level reply to a top comment, and so on.
I know HN does something similar, but it is not quite as good as reddit. From observation, specifically the 'penalty', or added performance requirement, of latching on to a top post is too weak. The result is that all HN comment threads consist of only a few top level posts, with subthreads growing off them, because you can easily 'jump the queue' just by commenting to a top comment. This is also what contributes to the idea that it is pointless to make new root level comments after an hour - because almost all the action is in subcomments to top comments.
Edited to add: Reddit soft-hides (collapses) subthreads that are deemed lower importance, which is probably key to make the ranking system work. Anyone interested in a subthread may expand the hidden/collapsed sections, and they may even be upvoted back to uncollapsed state. But by default they don't muscle into the main conversation. HN already has the collapse feature, which could be reused for this. It's just a client-side collapse, also the reddit one (though in huge threads, deeper threads will be loaded on demand).