You need 500+ karma. After that, a downvote arrow appears right under the upvote arrow.