It seems that flagging is as much "I don't like this" as "this is not appropriate".
Likewise downvoting. Maybe instead display both downvotes and upvotes, so people can see if a comment is found controversial?
Publishing upvotes and downvotes would allow others to see these effects, and would allow them to register a comment as "controversial" rather than "not worthy of notice".
Data on who made what vote could also be useful.
1. I want to know who upvoted/downvoted/flagged a comment so I can identify attempts at censorship, revenge voting/flagging, etc.
2. I want the ability to exclude individuals from the upvote/downvote tally I see.
For example, if I see someone downvoting every post that is favorable to Elmer Fudd, or upvoting every post that is favorable to Bugs Bunny, I want to block that person's emotional bias from the rankings I see in order for my view of the comment section to be as objective as possible.
https://news.ycombinator.com/item?id=2403716
https://news.ycombinator.com/item?id=2445039
That is, in the old days the total score was shown to everyone, but it was deliberately changed.
It doesn't mention revocation of flagging permission for abuse, and I think that would be helpful.