The problem is that people would abuse the downvote button and use it as a representation of "I disagree with you" is read of "this comment adds no value to the conversation."
2- Why users with karma will have a better understanding of downvote than the passives old-regular-readers? Peoples who read HN every days love the high level of the comments here, let them have produce some feed back. Note that I just defend my interest: I'm not good at creating comment with value (so I do not comment a lot), but I feel confident to reconise good comments from bad comments.
They really should scrap the reply link entirely and move it next to the "flag" button e.g. | reply
This thread is an example of why the delay is implemented, and this comment is an example of low value comments.