I sometimes invite people to write such comments if they disagree with me. Often, when a comment makes a factual assertion for which I know of no evidence, I will reply to ask for the evidence. I will say things such as "Citations, please?" or "What are some examples of this?" or in some other way ask for the participant to back up the statement to which I reply.

But there are cases when a comment is totally wrong, and not a good contribution to the thread besides (which has always been a generally supported basis for downvoting), and it's just too much bother to post a countering reply or to ask follow-up questions. In such cases, I don't feel embarrassed at all in downvoting the wrong comment. The site founder pg, who announced news of changes he is considering in flagging options on HN in this thread, wrote 681 days ago,

I think it's ok to use the up and down arrows to express agreement. Obviously the uparrows aren't only for applauding politeness, so it seems reasonable that the downarrows aren't only for booing rudeness.

http://news.ycombinator.com/item?id=117171

Voting and flagging options today are still much as they were then, and so for the moment I don't think it's a bad idea to downvote for disagreement, particularly if a comment has other undesirable characteristics. I will happily adapt to a new system if the site management implements a new system. And I invite participants who disagree with this reply of mine to write a comment and explain why my argument is totally wrong, if that is not too much trouble to you.