I'm pretty sure it's based off of the configuration in the green gear in the top right of the iframe. You can inspect the ways it's calculated from karma, comments, etc etc
But it would be useful to know if I had up/down voted them significantly in the past.
A plugin like HN Comments Owl would be more useful IMHO.