I know downvotes can be frustrating, but please don't give in to the temptation to add things like that last bit. It's against the site guidelines (https://news.ycombinator.com/newsguidelines.html) because, as they say, it never does any good and makes for boring reading. It also just attracts more downvotes.
Of course, there will be some comments which do not merit a reply (non-substantive contributions or blatant trolling) - but, those comments should be flagged anyway. Maybe count flags as downvotes (if they aren’t already), so you could still downvote reply-less comments by flagging them.
It encourages replying to bad comments which is already a bigger problem than the sting of an unexpounded-upon downvote. 'No receipts required for downvotes' makes HN better.
Edit: Notice how I upvoted you rather than downvoted out of disagreement
Downvote me all you want, it won't silence my thoughts.