You can downvote someone if you have less karma, but you have to have at least 25 (I could be a little off on the exact number). Nobody can downvote submissions though.