You reverse causality, you shouldn't use a downvote when you disagree with someones opinion to begin with.

I took a karma hit of -4 on the comment above just because people disagree with an idea I proposed.