You get a downvote button, separate from "flag", once you reach a certain karma level (I forget the number, though)