( ) I don't agree with this comment
( ) This comment is factually incorrect
( ) This comment is offtopic
( ) This comment is rude or inappropriate
(Add and replace verbiage to taste, of course)
Then people's downvotes would be imbued with some semblance of nuance.
And get rid of the score, and make downvote categories have hidden negative values (so people are truthful in their answer).
Downvotes or not for nuance or dialog, that's what comments are for.
Downvotes are solely to demote content that is viewed, in the subjective opinion of the voter, as not contributing to productive discussion. If it deserves a nuanced response then it doesn't deserve a downvote.
Says who? It's not in the guidelines, so really when to use the feature is determined by its effects. Downvoting obscures comments and hurts a user's karma. So in reality, downvoting is to demote content you don't want yourself or others to see for whatever reason and/or to "hurt" (for lack of a better word) other users.
Your criteria of doing it based on whether it contributes to productive discussion fits in that, but it's not the only possible criteria users may determine for themselves.
mceachen's suggestion is to modify the effects to make the possible criteria space more narrow and more similar to the criteria you shared. When they see such a popdown, they'd have a better idea of what the downvote is meant to be used for. It would provide some guidance on the "proper" use of the feature.
I'm mentioning lobste.rs for the second time in this discussion, but lobste.rs has this and it works kind of well IMO, both to think twice when I downvote and also to try to figure out why someone was downvoted.
Then again, lobste.rs is a different community and there is a reason or maybe several why I hang both here and there :-)
HN's approach is to stick with something simple, accept that it has downsides as well as upsides, and resist the temptation to fiddle with the downsides by sacrificing simplicity. It's a strong temptation for a technical person to want to do that, of course, but there's a strong story why the downsides of downvoting are mostly intrinsic: disapproval stings. Most of what people say when they complain about the downvote system seems to boil down to that. That's not a technical problem. If there's something we can do to mitigate it, it would more likely come from helping the culture to evolve. I do think that's happened, and is happening, a little; it's just very slow.
You can't draw conclusions from how downvoting works on smaller sites to what HN should do. With online communities, size is the dominant variable. When there are order-of-magnitude size differences between communities, that is what explains why things work differently there—not subtle software design choices. Similarly, one can't draw conclusions from how HN works to larger sites like Reddit. One of the nice things about HN not having to try to grow ambitiously (it has grown linearly at more or less the same slope for many years) is that we can focus on being the best HN-sized-thing we can be. That leads to subtler, more qualitative kinds of growth.
A couple more points that bang around in my head when issues like this come up...
One form of complexity that's particularly important to resist is adding metadata and creating metasystems in order to compensate for things in the core system. The magical charm of all things meta gives that step a perennial allure. But it's often a mistake. If you go meta to handle something that isn't truly orthogonal to the core system, you end up with more of a mess. I think we have a case of this here. For example, when a comment is wrong, the way to address that is to reply with correct information. Why do we need a popup with a "wrong" selection? The core system—discussion threads—can already accommodate this function; we don't need a metasystem for it. Of course it doesn't accommodate it perfectly, but the imperfection is not for technical reasons, it's because people don't always agree about what's right vs. wrong, and so on. A metasystem doesn't fix those things, it just recreates them on another level. And now, as they say, you have two problems, plus the problem of how the two interfere with each other.
It gets worse. With any system, there are only so many complexity cards you get to play. Each time you play one, you lose the chance to use that card for something else. In the software world, often people don't know this and because they're under pressure, a system blows through its complexity budget—plays all its cards—near the beginning of its lifecycle, depriving it forever of the chance to be coherent or tractable as it grows. This is the root of the fatalistic complaints about software bloat that commonly come up in HN discussions, as well as the constant yearning to create new systems rather than be stuck, as most of us are, fiddling with a couple screws in some rusty component of a massive machine that nobody understands. It happens that, from a technical point of view, HN is a rare exception to this pattern. For a bunch of reasons, two of which were pg's minimalism and Lisp sensibility, and another of which was perhaps that the project has always been resource-constrained, HN escaped the fate of becoming more complex than it needed to be. Preserving that quality is a priority, because it allows us to work with the system in rich ways that are impossible on most production software projects.
Less tongue-in-cheek, I do think flag/downvote reasons are worth the added complexity for us, but as there's a cultural component to these design issues it's probably a non-starter for HN to adopt. A change in such a fundamental interaction on the site would be an upheaval that's not justified by probability and size of an expected benefit.
+1 on basically your entire message, especially recognizing the limits of mods and size as dominant variable on roughly everything to do with communities. Best of luck.