Can we add that you shouldn't downmod someone simply because you don't agree with someone has to say? Downmods should be used when breaking a rule in the Guidelines.
For eg. opinions that you may strongly disagree with, but that you think anybody clicking the comment thread must read should be voted up so they appear higher.
Karma shouldn't be seen as keeping score on who is right or who is wrong, more about being on-topic and interesting.
http://news.ycombinator.com/item?id=2408048
In the final iteration of my suggestion, the upvote was replaced with the link word "interesting" for exactly your reason.
I imagine there is a karma threshold that unlocks this, but I couldn't say what it is.
It is very possible that flags are invisibly counted and this also factors into the ranking algorithm.
Obviously karma is publicly displayed, but posting just for the dopamine buzz of numbers ticking up should surely be discouraged. That's what WoW is for. Or even better, the gym :)
The reason for this is because, even though I might disagree with somebody regarding a certain topic, I may very well learn something new from this person, and, why not, even change my previous views on the subject. In that case, his comment was valuable and helpful, and in my opinion, should be displayed even more prominently instead of being downvoted.
EDIT: One more thing occured to me just now: The reason why upvoting/downvoting shouldn't be used to express agreement/disagreement is because it promotes writing comments that appeal to the general public, disregarding if they are actually right or not. This leads to writing comments that you know people will like, which makes commenting on some topic a sort of a popularity contest. In the end, we will have lots of comments that say what we want to hear, not what we (maybe) should hear. Perhaps I didn't phrase this as best as I could right now but hopefully the message got through.
Exceptions should be made when someone is presenting provably false information, but then a reply with counter-evidence is always preferable to a downvote. Downvoting without a reply is IMO only really appropriate for content-free comments that add nothing.
However, I prefer to disagree adding to the discussion.
I am downvoting you not because I disagree with you but because your statement was somewhat troll-ish. Don't take it personally. We should be nice to each other and disagree in a civil way.