Voting up or down is a nice, efficient mechanism for participating in a discussion when you do have a response, but it falls below the threshold of having enough time, enough interest, and enough worth saying to justify an explicit post. There's nothing illegitimate about responses that happen to fall below that threshold. In aggregate, they add great value to the site. To try to force people to make explicit comments that aren't naturally above the threshold would spread tedium, not civility.
People sometimes allege that opinions are being downvoted to oblivion on HN merely because they're unpopular. In my observation, this is rare. There's almost always some obvious reason (rudeness, irrelevance, etc.). But it's more self-flattering to think that your brave independent thinking was run over by a mob.
Egregious arbitrary downvoting does occur, of course, but it tends to get corrected. When I run across a comment at 0 or -1 that I see nothing objectionable about, I upvote it. Lots of other users do that too, so controversial comments tend to stay roughly at par. Lightweight self-correcting systems are hard to come by, so I think it's worth recognizing when you have one. That doesn't mean that experimentation is a bad idea, of course, but it will be hard IMO to come up with a better voting system. (The flagging system is a different matter.)