A ten minute conversation with the average voter will repel you from the idea that eveyone should want to listen to what 51% of most people would upvote, or should dislike what 51% of people would downvote. But your frontpage will be filled with the former, and you'll never be able to find the gems in the latter without serious time investment.
Upvote isn't enough. Hence, moderation.