One of the issues is that places like Hacker News and Reddit use an aggregate vote count. You can't tell if a comment that has a score of -10 is a comment where ten people saw it and decided to downvote it, or where 100 people saw it, 55 downvoted it and 45 upvoted it. This leads to situations where it can seem like a community rejects a point of view, even whe the point of view is shared by almost half the community.
Further exacerbating this is that, with an infinite number of votes allowed, someone who is less restrained about their votes is going to have a greater impact than someone who reserves them for only exceptional comments. For example, say 60% of the community likes Coke and 40% likes Pepsi. However, the Coke supporters are relatively calm and only occasionally vote for comments about soda, while Pepsi supporters are zealots who will downvote any comment that's supportive of Coke and upvote any comment supportive of Pepsi. A user is just going to see a lot of highly downvoted/greyed out Coke posts and a lot of highly upvoted Pepsi posts, and assume that the community is highly supportive of Pepsi while hating Coke.
That's more of a problem with voting in general (and a larger problem with how time = speech on the internet), but downvoting can certainly exacerbate the problem.
Upvote only, it rewards the best opinions and analysis.
The problem is, botnets and astroturfing use downvotes to manipulate conversations, perceived word of mouth and ultimately the algorithms.
For instance, since Youtube has removed downvotes from visibility, my recommended videos are more in line with what I want. I think the downvote was a massive tool for shaping the algorithms and recommended videos that becomes something that isn't organic.
I downvote only the downvote. Down with the downvote!