Based on how much some of my comments bounce up and down, I don't think that the downvotes are done by only a few. But I think this is a good thing, and if anything we need more people eager to downvote poor quality submissions. I think the reason that HN has remained as usable as it has because it is actively policed for content quality.
I don't see a way to check, but I'd guess that my upvote to downvote ratio is about 3:1. I try to vote up anything that would want to see more of on the site, whether I agree or disagree with the content. I try to mercilessly vote down snappy one liners, purposeless profanity, and dumb jokes. If anything, I'd say I vote primarily on form --- I'll support just about any content if it's well written and well intentioned.