Comment by barry-cotter | Hacker News Reader

Comment by barry-cotter | Hacker News Reader

Parent Full thread

Yes, peer review should be an informal process that happens after publication, not a formal one that turns a 40 page paper into a 100 page one and takes six years.

Many fields have developed work arounds to peer review in the form of working papers, preprints and conferences. That this is necessary suggests that peer review be burned to the ground and the ashes sown with salt.

https://www.stat.cmu.edu/~larry/Peer-Review.pdf

> The refereeing process is very noisy, time consuming and arbitrary. We should be dissem- inating our research as widely as possible. Instead, we let two or three referees stand in between our work and the rest of our field. I think that most people are so used to our system, that they reflexively defend it when it is criticized. The purpose of doing research is to create new knowledge. This knowledge is useless unless it is disseminated. Refereeing is an impediment to dissemination.

...

> Some will argue that refereeing provides quality control. This is an illusion. Plenty of bad papers get published and plenty of good papers get rejected. Many think that the stamp of approval by having a paper accepted by the refereeing process is crucial for maintaining the integrity of the field. This attitude treats a field as if it is a priesthood with a set of infallible, wise elders deciding what is good and what is bad. It is also like a guild, which protects itself by making it harder for outsiders to compete with insiders.

You’re creating a false dichotomy. It’s all about maximizing signal to noise, and expert reviewers are absolutely required (at some point in the process) regardless of whether they’re perfect.

What false dichotomy? If you want to keep journals revert to editorial review. If the editor thinks the paper isn’t garbage they publish it. Sometimes they’re wrong but you don’t get papers published six years after they’re written.

Maximizing signal to noise is a terrible goal to put so much weight on. We should be trying to maximize the growth of knowledge, not minimizing the number of wrong papers, which is what max(s/n) leads to.

Follow the editorial philosophy of Max Planck editing Annalen der Physik during the period when that was the pre-eminent journal in Physics during the most productive epoch for Physics research ever.

> To shun much more the reproach of having suppressed strange opinions than that of having been too gentle in evaluating them.

> We should be trying to maximize the growth of knowledge, not minimizing the number of wrong papers, which is what max(s/n) leads to.

IMO, filtering out noise is a big part of maximizing the growth of knowledge. I cannot read every paper in my area that goes up on Arxiv, let alone every paper in related areas. That's why it's useful, albeit imperfect, to have gatekeeping peer-review organizations to force experts to look at papers and pick out the ones they think are most useful.

There is a lot of noise on Arxiv.

> That's why it's useful, albeit imperfect, to have gatekeeping peer-review organizations to force experts to look at papers and pick out the ones they think are most useful

Gate keeping and review do not require anything like the current peer review system. The current peer review system retards the flow of knowledge, or it would if anyone relied on journals to know the state of the field.

Gate keeping can be done by editorial review. If you really want to you can have accept/reject only with no revise and resubmit cycle like Sociological Science. They aim for 30 day from submission to publication.

Refereeing wastes more than $50m a year in value in Economics alone. Burn the system to the ground.

> IZA DP No. 12866: Is Scholarly Refereeing Productive (at the Margin)?

> In economics many articles are subjected to multiple rounds of refereeing at the same journal, which generates time costs of referees alone of at least $50 million. This process leads to remarkably longer publication lags than in other social sciences. We examine whether repeated refereeing produces any benefits, using an experiment at one journal that allows authors to submit under an accept/reject (fast-track or not) or the usual regime. We evaluate the scholarly impacts of articles by their subsequent citation histories, holding constant their sub-fields, authors' demographics and prior citations, and other characteristics. There is no payoff to refereeing beyond the first round and no difference between accept/reject articles and others. This result holds accounting for authors' selectivity into the two regimes, which we model formally to generate an empirical selection equation. This latter is used to provide instrumental estimates of the effect of each regime on scholarly impact.

https://www.iza.org/publications/dp/12866/is-scholarly-refer...

I agree that the multi-year publication timelines for journals are in general silly. I agree that keeping up with a field by reading journals doesn't work. If by editorial review you mean "the editor alone decides", I don't think that's a good replacement.

I think the computer science publication model of a few conferences per year with ~3 month submit-to-decision timelines, based on 3-5 reviews per paper, is pretty good. But it's probably not appropriate for fields with deep contributions that take substantial time to verify, like math.

Reply on news.ycombinator.com