You can't downvote comments if you have less than 501 karma, and the downvote button doesn't appear on replies to your own comments. Plus, there is no downvote button for stories, just for comments.

EDIT: It originally said "less than 500 karma".

not enough karma points.