Comment by dahart | Hacker News Reader

Comment by dahart | Hacker News Reader

Parent Full thread

I’d agree with all those thoughts with respect to the system we have today. Yes, the goal of de-anonymizing reviews would be to eliminate trust in reviewers as a requirement for achieving impartial and fair reviewing (given the many ways we’ve already seen trust failure), and hopefully the side effect would be to increase overall trust in the system. Making reviewers part of the public record would also allow exposure of bad behavior and explicit collusion, without having to go to great lengths to prove it like they did in the article. I’m not entirely sure that the level of candidness that anonymity invites is necessary or helpful for a healthy and robust review system. But, note that my suggestion above doesn’t depend on critical feedback at all. One of the things I’ve found problematic in today’s review system is that reviewers assign absolute scores to a paper, but every reviewer has their own notion of what is good and bad. It’s common for an entire group of reviewers in one sub-topic to give average lower scores than a group in another sub-topic, meaning that different sections of a given journal or conference have different acceptance rates. My suggestion for rank choice voting is partly to ensure that all reviewers are working in the same units, and are all weighted equally. (But I assume there is a possibility of unintended consequences and/or opportunities to game the system with what I suggested, so I’m curious if people see ways that might happen.)

If reviewers are de-anonymized, the opposite of what you think will happen. Nobody other than the most socially stunted would be willing to write a harsh review and make an enemy for life. People are petty. Most fields in academia are incredibly small, with everyone knowing each other. Reviews would become way more positive and useless. Collusion for positive review is pretty rare and wouldn’t really change with real names anyway because it’s pretty subjective to claim a positive judgement is obviously the result of collusion. It’s also already the job of the editors to collate the reviews and make his own judgment based on them. If one review is overly positive and adds no useful information, it can and will be discarded. The editor can even call for another reviewer if he/she is still undecided. I’ve reviewed papers and recommended acceptance, yet the paper was still rejected; the opposite has happened too. As a reviewer you are only advising the editor.

> Reviews would become way more positive and useless.

In my corner of CS, there are plenty reviews of debatable use that are very negative. In fact, I have a pet hypothesis that the average score of the lower scoring but accepted papers is negative (scale from -3 to 3). And I wouldn't be surprised if the median paper's score was negative.

Positive uselessness doesn't seem like that much worse.

Are you sure about that? I’m not sure you are hearing and understanding my suggestion. A review would always consist of one paper being ranked above another one. The tone of the review comments would have no bearing on the usefulness of the review. The reason we even have critiques in reviews is to justify the score that the reviewer assigns. Reviewers are specifically asked to produce reasons. What I’m suggesting is different because reviewers would rank two papers against each other and not be expected to justify their decision, nor would they be asked to provide a magnitude for their opinion. Comments and critiques would be allowed, but there’s no need for harsh reviews.

I’ve seen a lot of overly and unnecessarily harsh reviews. Anonymizing enables over-stating criticism, it happens routinely. I don’t think I agree that harsh critique is necessary for a healthy review system. It is already the case that good reviews are not extremely harsh, they focus on the facts and are willing to stand by their statements. I don’t personally know that many researchers who have trouble being direct in person and face to face, or of offering constructive criticism.

> As a reviewer you are only advising the editor

This completely depends on the journal or conference. Quite a few of them, especially the larger ones, do not override reviews casually nor often. And what I’m suggesting is a system where this idea can change, where editors can more easily trust the review results, and won’t need to override the decision.

> People are petty.

This might well be true. And so I’m not entirely understanding your argument. It seems to be simultaneously suggesting we have a problem, and defending the status quo as the way it needs to be. What would you suggest as a way to improve the review system so that pettiness has less influence than it does today?

Ranking only makes sense in a venue where the number of papers or presentations is limited. There many online journals of good reputation that will publish anything over a certain (obviously not perfectly well-defined) bar. The number of papers tends to increase over the years as the numbers of researchers also increase, and this is considered acceptable. Other journals are so selective that they won’t publish anything if nothing is deemed worthy.

You’re also misunderstanding the point of reviews. They also serve as comments to the authors to modify their manuscript and make it acceptable for publication.

Reply on news.ycombinator.com