It's because those who agree rarely bother to upvote, while those who disagree downvote often. It's the same in real world: disagreement in form of protests is loud and visible, while even massive support is quiet.
One fix HN could make is to take into account the number of readers before hiding a comment: 4 downvotes is big when there are only 10 readers, but it's nothing when there's a million.