I've been here long enough to see this argument before, yes.
I feel the same as I do now: that if it looks like people are using the downvote as an "I disagree" or "I don't like you" button, people are entitled to call this out and ask for an explanation. If no good explanation is forthcoming, this suggests the downvoters might be in error.