Nobody can downvote stories, but if they're violating the rules, high-karma users can flag them for deletion.
Fortunately, people haven't been doing a lot of gaming of the system yet. The biggest problem with upvoting only is that controversial posts garner the most votes (a thousand people love a subject and ten thousand people HATE it, so it gets a thousand upvotes and a TON of comments, even if most people disagree with the premise).
I repeatedly think that I would like downvotes, but I quite quickly remember that I absolutely hate downvotes. I don't completely agree with being able to downmod comments, however I think the side effects are dampened that you have to have good karma before you can downvote, which prevents trolls from abusing the system.