This is an important point.
>It's not just downvotes but any votes that have negative social consequences.
Upvotes also tend to select for popular modes of thinking. In the end, a system of only upvotes makes people want to only post things that will be perceived as popular by the community.