I've also wondered about downvoting... I never see a button for that, but it seems like some people are still able to. Is it just the 'flag'?
Reply on news.ycombinator.com