Comments at -4 can still be voted down, but the displayed score will not go any lower. (I'm not certain of the rationale, but I think it's to somewhat discourage excessive downvotes just to make a public show of disapproval/disagreement.)
Why can't I vote comments down? I think it's bad form to kick a comment while it's down :)
Users with (I think) at least 200 points can downvote.