If everybody has the ability to downvote, then a big chunk of interesting comments never see the light of day, and a lot of interesting users leave, deciding they're not wanted.

I've disengaged from lots of subreddits because any comment that doesn't vociferously echo one of a half-dozen talking points immediately gets downvoted by teenagers who joined yesterday. It's a discouraging way to build an active/friendly community.