Hacker News Reader

Comment by ajarmst | Hacker News Reader

The part I found interesting was that the author notes that it is well-known that the quality of the work has little to do with its odds of acceptance even when the system is 'functioning' as designed. Is it cheating to game a system that is already fundamentally broken? If the quality of your work is inadequate to secure publication and your career depends on it, it wouldn't be very hard to convince yourself that you can't cheat a rigged game. What 'integrity' is actually being threatened? Perhaps some of the energy devoted to identifying these collusion rings would be better spent developing a review process that is at least somewhat biased in favour of good research rather than the density of the authors' professional network. Or at least in mitigating the poisonous 'publish or perish' rules that lead to this sort of phenomena.

View on HN

> the author notes that it is well-known that the quality of the work has little to do with its odds of acceptance even when the system is 'functioning' as designed

I think this is a little more pessimistic than what the piece says. The NeurIPS (then-NIPS) experiment said that about 60% of papers accepted by one PC got rejected by the other. That doesn't actually mean "the quality of the work has little to do with its odds of acceptance". It may just be that there's a paper has to cross a quality threshold, and once it's past, then the outcome has a lot more variation.

My personal take on NeurIPS specifically is that there's a fraction of bad papers, maybe 40%, that probably shouldn't and won't get in. Then there's a minority of very nice papers that probably should and will get in, maybe 5-10%. And then there are a bunch of middling papers where a lot of it is luck and drawing friendly reviewers. But these aren't bad papers, and you can't really just churn them out, they're just not very good papers.

Seems it's also rational for an opportunistic player of the publications game to split their work into the maximal number of marginally acceptable publications, which would result in a high rejection %.

I wouldn't be surprised if that's already happening and has become the preferred strategy for the last decade at least.

https://en.wikipedia.org/wiki/Least_publishable_unit

Don't forget there can be papers recognised universally as acceptable.

From memory, it was 25% rejected by both PCs, 15% accepted by both PCs, and the middle 60% random.

then why not review projects in a double blind fashion ? Those which are accepted twice will get published and voilà. But that's twice the work.

Double-blind usually means something else in reviewing: the reviewer does not know who the author of the paper is, and the author does not learn who the reviewer is. Single-blind reviewing is when the reviewer knows who the author of the paper is, but the author still does not learn who the reviewer is.

As you mention, the problem with having two sets of reviewers is that it's hard enough for conferences like NeurIPS to find one set of qualified reviewers. Usually at least 1/3 of reviewers on any given paper produce a poor review, either because they don't care or because they really lack expertise. Complaining about this is so widespread that it even doubles as a sort of icebreaker for researchers, but nobody has a good solution.

I agree with one of the other posters, the result of the study does not highlight a problem that the acceptance of the work is unrelated to the quality. It's more a fact of the distribution of the "quality".

Essentially if you look at review scores for a conference which has say a 40% accept rate (and this is quite similar across fields I'd imagine), you find there's 10-20% (depending on conference) of papers that are clear reject for all reviewers, then there's probably around 10%-15% of papers which are very clear rejects now the rest of the papers are very similar in scores so the cut-off becomes quite arbitrary (and depends on luck as well). This is actually well known for grant applications and a sign that there is likely not enough money in the system.

Reply on news.ycombinator.com