Downvoting is too blunt a tool. You can downvote for many reasons "(I think) you are wrong", "your comment adds nothing to the discussion (simply repeating the original)", "I don't like you (lets be honest that is some peoples reason)"
Also the suggestion would balloon the number of comments and make the thread even harder to read