I disagree. I see, and myself, do downvote comments that I disagree with, if it is a strong disagreement. I think there is also a post where pg says something similar: http://news.ycombinator.com/item?id=392347
I have found that comments of mine that get downvoted cause me to think about why.