The downvote button and flag button are separate buttons.

I believe the downvote button is unlocked at 500 karma.