I do not downvote for comments I disagree with.
I downvote for off topic, or for comments which detract from the conversation. I believe most others with the ability to downvote do the same.
Replying to off topic posts would be counter to the purpose of downvoting.