Comment by apu | Hacker News Reader

Comment by apu | Hacker News Reader

Parent Full thread

I'm all for new publishing models in academia, but this post doesn't address the fundamental difference between the web in general and academic papers: expert curation.

The web's "curation" works (to whatever degree it does) because there are thousands, if not millions of people who can rate the quality of most articles/essays/websites. But in many scientific fields, there are often only a very small number of people who would be qualified to review a particular paper (usually well under a hundred, sometimes as few as a handful).

And make no mistake, this is a numbers game -- there's the well known 100/10/1 rule for content websites (for every 100 viewers, there are 10 who rate content, and only 1 who creates content). The exact numbers vary quite a bit from website to website, but the point is that it's orders of magnitude difference. That's all fine if your initial pool of people is huge, but if you start with 20 qualified people...you see the issue.

Another related point not addressed is that peer review is fairly demanding work. You're not just saying "accept" or "reject", but you're obliged to write a page or two suggesting the strong and weak points of the paper, things to improve, etc. You need some mechanism for enforcing this (even if it's via social norms), otherwise review quality deteriorates quite severely, which in turn affects paper quality.

In lieu of this article, I'd recommend seeing lectures by (or essays by) Michael Nielsen, who's thought about this problem in much more detail and (I believe) has a much better understanding of the way things work.

http://michaelnielsen.org/blog/essays/

Thanks for your detailed thoughts.

Peer review is already evolving. Increasingly academics discover their papers via search engines (Google, Google Scholar), and social platforms (blogs, email, arXiv, Academia.edu, Twitter). The usage of these channels as discovery channels for research shows that the metrics that drive those channels are doing a good job at expert curation - i.e. surfacing good content.

On Google Scholar, it's citation counts that drive the ranking, and therefore the discovery process. This is the "crowd review" process I talked about in the post. With social platforms, it's recommendations from friends and colleagues that drive the discovery process. This is the "social review" process I talked about in the post.

In the pre-web days, people used to walk down to their libraries to read the latest edition from a particular journal. In those days, the journals did drive a lot of the discovery process. Nowadays, pretty much all discovery of research papers happens on the web, and most of the discovery is via channels such as search, and recommendations from colleagues over social & communication platforms.

You are right to think about the incentive systems that drive different kinds of peer review. In the case of the social review process, you want to share good recommendations with your friends, not bad ones (this is why Twitter and Facebook work as discovery channels). In the case of the crowd review process, you want to cite good works that your work genuinely built on, not bad work.

My understanding is that most publications are not heavily cited, and address very specific problems. I'm curious as to how you plan on creating incentives for qualified people to review the papers in their field which focus on problems outside of their present research interests.

To give a personal example: I used to work in a research group that applied techniques of a field called geometric control theory to the problem of modeling flocks of birds. There are not that many geometric control theorists in the world, and their interests range from robotics, to circuits, to modelings schools of fish. Right now, there are journals of control theory that force their reviewers to review all submissions, regardless of their personal interests.

How do you plan on emulating that? In other words, there need to be incentives for geometric control theorists who work on modeling schools of fish to spend a few days of their time grinding through the proofs of some grad student's paper in modeling bird flocks, which he needs for his thesis.

Peer review often works the worst when a journal asks someone who's not that qualified to peer review a paper. See the comment below from billswift about an unbelievable case of peer review going wrong, where the reviewers weren't in the research area of the target article (you need to follow the links to get the full picture).

Something doesn't have to be heavily cited for it to start getting traction, just as a website, or blog post, doesn't have to get heavily linked for it to get some traction. A friend might find the paper and share it via Twitter, their blog, or Academia.edu; this is how the social review process works on Twitter and Facebook. It might sound slow: you might wonder 'won't it take ages for a paper to get traction like that?'. But in reality, when you have social platforms like Facebook and Twitter, content can be surfaced, and whip around the world, at incredible speeds. The journal peer review process, by contrast, is several orders of magnitude slower.

The web is known for having a long tail of content: lots of content that appeals to certain niches of people. It is part of the magic of the web's discovery channels that this long tail content gets routed to the niches who care about it - via search, Facebook etc. The fact that academics overwhelmingly use the web now for research discovery is testament to how good a job the web's discovery channels are doing at surfacing good content in highly niche areas.

As a researcher myself, I do not use the web as a means of content discovery. I do use google scholar, pubmed, and citeseer, but these are only useful because they restrict the search to just peer reviewed journals (and optionally patents in google scholar case).

Trying to use general web search is often far too noisy. Often times I find interesting papers on page two, three, or four of google scholar. In a general web search these same papers might be on a much later page, or perhaps buried so far that they don't come up at all.

As others have said, there is also much more to peer review than discovery. Peer review is additionally intended to help authors improve work that isn't quite up to standards. Academics do peer review for free because they recognize the value of it and because they are asked directly by the editors to do it. In your model, what would be the incentive to look for new papers and give reviews? How would you handle old versions of papers with mistakes or papers that are of insufficient quality? The peer review process currently filters these intermediate stages of paper writing. In your proposed model, you would potentially have many versions of the same paper that a user would then have to filter through, in addition to a very great many papers that are of very low standard.

The quality of content on the web and its curation is a very poor standard to compare academic literature to. Many users are happy enough to be able to filter out obvious spam pages let alone judge quality to the level needed in academics.

It's true that the papers in the indexes of Google Scholar, Pubmed, and Citeseer are peer reviewed, but what that shows me is that what is really driving the discovery process is the ranking system in those search engines: i.e. the order in which the hundreds of results for a given search query show up.

One of the drawbacks of the existing peer review process is that academics don't get credit for their reviews, nor do the reviews see the light of day for others to benefit from. I expect that there would be significantly more discussion, and reviews, of papers in the future if there was a credit system that allowed people to get credit for reviews and comments they made of papers. I think that credit system is possible and will be built.

There is an interesting question regarding the immutability of content. Right now, once you publish a paper, you can't edit it, or delete it. It's an immutable piece of content. Before the web was established, there was a line of thinking, developed by Ted Nelson, according to which the internet should evolve like that, and that a link should always work: once some content is posted, it can never be taken down. Most people are probably glad that the web developed along the lines of Tim Berners Lee's thinking, rather than Ted Nelson's, and that they are now free to edit and delete content they have posted. I think similarly people would appreciate being able to update a paper in response to a comment they have received. The author is better off, and so are subsequent readers, as they find themselves reading a more evolved and advanced paper.

Reviewers are intentionally hidden from view. The reason is to prevent social or political backlash from a poor review. In fact, it is sometimes the case that papers reviews are double blind, the reviews do not know the names of the authors either. This is also to prevent bias on the part of the reviewers. This secrecy is a positive and important part of the peer review system. Science should not be politically or socially biased.

The immutability of published works is also crucial. It routine for writers to leave out details covering in prior works. This saves immense amounts of time on both writers consumers of scientific works. However, it also means that it is crucial that all cited works be preserved forever. If a document truly goes missing, then entire lines of work become incomplete. Papers can and are infrequently withdrawn, but as far as I know, the work is not erased, but merely marked as bad.

The way updates or corrections are made is via newer papers revisiting topics. But it remains that at every step some amount of decent due diligence is done to correct errors and not clutter up the records with incomplete versions.

As great as the web is for unstructured content, you cannot easily apply it to every area, and especially not to scientific publications. There are plenty of other examples of curated sources on the web that crucial. Map systems, curated databases of restaurants, directories of people like LinkedIn and Facebook, and even Wikipedia can be counted a curated system due to its system of editors.

Scientists have always and are still free to share data, white papers, and whatever else outside the peer review system. The main reason peer review is still here is that no suitable alternative has ever been proposed that addresses all the points that peer review does. There is a push towards open access journals to benefit the world at large though.

I understand that you are passionate about this, but I'm not convinced by your arguments.

Reply on news.ycombinator.com