What upvoting is supposed to be: "Good post that adds a lot to the discussion." What it really is: "I agree with this guy."
What downvoting is supposed to be: "Person is spamming, making a weak point or arguing in an inflammatory manner." What it really is: "You're wrong shut the hell up."
This problem could be fixed if people were forced to give a reason for their upvotes and downvotes. Imagine if every time you click upvote, a small textbox appears that has a list of predefined options, such as "valid point" or "good contribution" or "i learned something". Similarly, every time you click downvote, it automatically takes you to the Reply page; if you don't reply, your downvote does not go through.
This kind of transparency would encourage people to be more thoughtful when they cast their votes.