Book Review: Making Nature
astralcodexten.substack.com
astralcodexten.substack.com
Things are already changing, but very slowly and almost imperceptibly if you look at it day-to-day: Public research repositories like arxiv and biorxiv, along with public code repositories like github and gitlab, are becoming or maybe already are the world's most important academic "journals."
All research and code posted on them gets a quick once-over; good work gets the attention it deserves; bad work is quickly ignored. Reviews take place over the Internet via both public and private forums.
Gatekeeping power lies more and more in the hands of a global, distributed scientific community open to anyone willing and capable of doing and reviewing the work. It's fabulous IMHO.
In my experience, I've noticed a few things that are potential problems:
1. I was surprised by this, but Google seems to delist papers in its search results after a certain amount of time. People have complained about older results vanishing from Google search; this seems to apply to archive papers as well. I wonder if there's space for a dedicated academic search engine (although to be effective, such an engine would have to index regular web sources as well).
2. People still use peer review as a yardstick, for better or worse. So you'll see something reported in a mainstream journalistic source, with a caveat that it hasn't been peer reviewed yet. It would be nice to have some way of implementing peer review in a more distributed way. There's been a fair amount of thought about this, although it seems like efforts to do so kinda lost steam after archives started taking off; it's almost as if there was a collective "ok we're exhausted and have managed to get archives some traction so we'll just leave that peer-review problem on the table".
3. It seems like there's some room for other types of archives other than the "-arxiv" sites. I know they have been out there but they've not attracted quite as much attention.
4. Journals do provide copyediting services; it's a relatively minor thing but professional layout, fonts, and design do improve readability.
I think the second problem is the most pressing, maybe along with the first.
It's a little odd to see a paper in a public repository getting a lot of discussion from experts in the field, approving it essentially, and then it getting treated as second class because it hasn't undergone formal peer-review. Sometimes the blog posts and social media discussions are more thorough than what would ever happen in a formal peer review context. On the other hand, I still think anonymity is important in peer review, and social media hype has its own set of serious problems.
Yes, it would be great to see open archives become the standard. As the OP points out, for most of Nature's history, it did not have any formal peer-review process; Nature was quite literally a journal for sharing new research faster than via publication of a book, which was the previous dominant method for disseminating research. During most of Nature's history, peer review was something that happened organically, informally, in parallel. For most of its history, Nature was in fact used in much the same way that open archives are used today!
I also agree with you on points 1-4. One possible solution to the problem you raise on point 2 is for the open archives to implement an open peer-review system -- although it's really hard to do that well at scale. (HN moderators nod in agreement.) Hopefully the folks at the various open archives are already thinking about peer review too.
The attempts that come to mind are:
(1) I think there's some service that links papers to pubmed or something, to "attach" comments to the papers?
(2) Some arxiv services have allowed for "endorsements" by other authors or users.
My experience with both of them is that, deserved or not, there's a sense that it's hard to interpret them because of issues related to the informality of it all and lack of accountability. I personally sort of like the second option but it's fairly low-effort and hard to interpret that way.
As I'm typing this my sense is that at some level what people are really looking for with peer review is some editorial process that people have gone through to reach some threshold. In both of those cases, the peer review comes after the paper is written, and there's not a lot of incentive to improve it or change it except through the authors' own conscience.
Of course, we've seen how the traditional peer review system fails or can be abused, so how to achieve that threshold of "this paper has gone through some kind of editorial revision process" without it turning political or random is the tricky part. I guess it's a way of figuring out how to have one's cake and eat it too. Maybe some system where authors can choose whoever they want to have edit it? That's susceptible to extreme abuse but maybe not more overall abuse than the current system.
I personally am fine with commentaries and endorsements -- I'm not sure why people can't think for themselves -- but I do think as you get into technical areas it helps for someone outside the field to know a paper has been looked over by someone else knowledgable. It just seems like we've gotten way far away from that original goal.
The link to 1971 charts is intriguing. But the link seems to be on a gold standard/Bitcoin web site. Is this a point of conversation among, ahem, boring mainstream economists?