I'm pretty sure that downvotes from some users, perhaps those at a given karma threshold, count twice. I've seen quite a few posts that aren't otherwise attracting a lot of attention go straight from 1 to -1 more-or-less instantly, or from 3 back to 1.

It's also possible that they have experimented with fuzzing votes like Reddit does.

Either way, they don't seem inclined to disclose how it works behind the scenes. The Kremlinology of Hacker News is half the fun.