Have you ever thought about a rule, that you can only downvote a comment if it has at least one reply (by anybody, not necessarily the downvoter)? That would discourage the demoralising situation where someone gets downvoted but is left guessing why.

Of course, there will be some comments which do not merit a reply (non-substantive contributions or blatant trolling) - but, those comments should be flagged anyway. Maybe count flags as downvotes (if they aren’t already), so you could still downvote reply-less comments by flagging them.