Hacker News Reader

Comment by MikeTaylor | Hacker News Reader

[Note: I am the author of the original article, so I have a horse in this race.]

I agree that the journal system provides a valuable filter. My beef is not with journal, but with locking up publicly funded research. Open-access journals are an unequivocal good.

But I am not convinced that all journals filter on the right things. I like the approach of PLoS ONE (http://www.plosone.org/), which assesses papers purely on the quality of the science, and ignores subjective notions of "impact" or "importance". The result is that everything they publish is good -- as reliable as articles published in any other journal -- but it includes articles that would never get into, say, Science or Nature due to the self-consciously super-selective approach those journals take.

Anyway, that's a side-issue. The real issue is that, whatever selection criteria a journal uses, it should make the resulting papers freely available to the citizens who funded them.

View on HN

>which assesses papers purely on the quality of the science

Mike,

1. Thank you for taking up the cause.

2. Trust me: Boycotting reviews is easy in my case. Unfortunately, this means no change because I stopped doing them -- it was just taking too much time for no reward.

3. wrt the above quote: I note that there is a market for selective journals. Much like reddit and HN, it is useful to have a crowd-source selection of interesting/important items. (Yes things get missed, but S/N is much higher than if I had to do it from scratch.

Good luck.

> there is a market for selective journals. Much like reddit and HN, it is useful to have a crowd-source selection of interesting/important items.

We don't need journals for that. They could be replaced by services like http://f1000.com/

I was more envisioning a reddit/HN segmented by science/engineering disciplines.

In the current journal system, editor-votes have absolute veto power, and reviewer votes have large upvote power. This has led to much progress, but it is unclear that it is the best system possible.

Then there is the whole segmentation question, which is a big tradeoff between S/N and coverage of the space. An example: Physical Review A, B, C, D & E used to be one journal -- Physical Review. Phys Rev had a green binder, and people would keep these dead-tree objects on shelves for reference. Well, the publication volume kept growing: Some wag calculated that rate of growth of the volumes on the shelf would cause their edge to exceed the speed of light. (He also pointed out that no information would be exchanged, so there was no violation of physics.)

Interesting idea to use something like a subreddit as a place to communally filter ("post-publication peer-review") published papers. The problem is, you wouldn't want one-man-one-vote. You'd want established professionals in the field to have a greater upvote/downvote weight than J. Random Ligger. You could imagine a sort of pagerankish scheme where people whose own publications have been upvoted get more more voting power as a result.

> The problem is, you wouldn't want one-man-one-vote.

I agree. Right now the reviewers (= designated experts) have all the votes.

As I think about it, the reviewers achieved that status because they wrote articles that got upvotes from previous reviewers. The first reviewers were historical and they set the standards.

You could end up with a situation where the reviewers were poorly chosen and so upvote complete bullshit (http://en.wikipedia.org/wiki/Sokal_affair), but fields of science have a reproducibility clause that provides good amounts of self-correction.

Hmmm. I didn't think your boycott post would lead to an examination of the way science and engineering is published, but the time might well be ripe. That said: My recommendation is to narrow the scope and focus on the boycott. It's enough to keep your plate full for a while.

You could use some kind of pagerank, in which a vote for a journal is a vote for the academics behind it, increasing their vote power.

This would give more sway to old, established academics, but that's already the status quo (and roughly what you are trying to achieve anyway).

>but it includes articles that would never get into, say, Science or Nature due to the self-consciously super-selective approach those journals take

I think the main issue of "traditional" journals is that they are artificially constrained to publish 10 to 15 articles per month (or issue), whereas electronic publication can get rid of this constrain (as long as they have enough peer reviewers).

Everything PLoS ONE publishes is good because they are less selective than the top tier journals? Seems an extraordinary claim to make.

No, everything PLoS ONE publishes is good because they reject everything that isn't good.

The "less selective" bit manifests itself as a reduced average "impact"/"importance", rather than as a reduced average quality.

Exactly. Paradoxically, this policy has resulted in its getting a rather good impact factor for such a new journal (4.411 for 2010), which happily attracts more high-profile submissions in a virtuous circle.

The moral seems clear: other things being equal, freely accessible papers tend to get cited more often than paywalled ones. (Who would have guessed?)

Perhaps I wasn't clear. Everything PLoS ONE publishes is good not because they are less selective, but although they are less selective. The reason their acceptance criteria don't adversely affect the quality of their published articles is because they do select on quality -- if it's not good, it doesn't go in. But they don't select on impact.

Ah, my mistake, I see what you're claiming. However, let me ask a more substantive question. You end by stating:"The real issue is that, whatever selection criteria a journal uses, it should make the resulting papers freely available to the citizens who funded them."

The OA model largely involves shifting the cost from the user to the author. This will be paid out of the authors grant money, which in many cases is publicly funded as well. It's not at all clear to me that the end effect is as different as the proponents claim.

Reminds me a bit of how Google uses "open" as a competitive weapon to push Android. I wonder if OA publishers are not doing the same to push their business model...

What exactly are those costs? Reviewers work for free. The only other thing that's needed is a web application and some hosting. These costs are negligible compared to the tax that closed journals are imposing on the scientific world.

I don't really know the details. But consider what open access journals charge per article: http://www.plos.org/journals/pubfees.php

This is not cheap, and I'm very glad I didn't need to cover this back when I was publishing papers.

If anyone knows a good article on the detailed economics I'd love to see it.

At a premier journal in my field, a "managing editor" is paid for the 25+ hours/wk of work assigning submissions out to the SEs/AEs (based on area of focus), and—more importantly—following up with them on their progress. As I understand, he's also responsible for managing itineraries and speaking engagements for the editor-in-chief, who travels regularly evangelizing (and getting feedback on) the top journal in our field.

We have a couple of EICs for major journals at our school, and I suspect (but do not know for certain) that the journal subsidizes their conference attendance too, so that the journal's senior editorial staff can meet in person at least annually.

My point is: don't constrain your cost focus to just distribution. There's more to running any organization than initially meets the eye.

Is all of that necessary? Would a model like the following work:

Have a web site where scientists can submit papers. Other scientists can then review the papers. The reviews are publicly (though optionally anonymously) published along with the papers, and the authors of the paper can submit revised versions of the paper.

This would not work. Reviewers would not feel comfortable being completely honest if they knew their reviews would be published publicly. In any case, there should be no need for the reviews to be public.

Even today reviewers run the risk of having their reviews made public, since nothing prevents the author from disseminating it. They are however normally anonymous.

Why should the reviewers feel unconfortable? In the current system, reviews are made available to the paper writers, and that doesn't seem to inhibit the reviewers, despite the fact that one often has a good idea of who the reviewers are, especially in the more niche fields of science.

Reviewers should be prepared to stand by their reviews. If they're not prepated for it to be known that they said something in a review, then they should not say that thing. Anonymity and secrecy in reviewing doesn't help the field.

No.

The website would get far too many terrible papers.

Many papers would get no reviews.

Many reviews would be terrible, and done by non-trustworthy people.

The problem with the system you are suggesting is that it feels a lot like wikipedia, and who "wins" on wikipedia has little to do quality, and a lot to do with who is willing to spend the most time editing / wiki-lawyering.

Sadly, the evidence so far strongly suggests that you're right, and this idealistic approach won't work. That evidence is from PLoS ONE (which remember is the main example of how many innovations do work). They have had the ability for people to comment on and evaluate their papers for a couple of years now, and it's hardly ever used.

So can we get the incentives so that scientists do review in such a system?

The EGU journals work this way. Official reviewers are assigned, but third parties can comment. An example submission

http://www.the-cryosphere-discuss.net/5/1749/2011/tcd-5-1749...

A hybrid model with both assigned reviewers and public comment is being tried by a regional journal (Southern Association for Information Systems): http://knol.google.com/k/craig-vanslyke/jsais-review-process...

Good point: But the analogy may/may not hold if we look at actual numbers. My bet is on it not holding and PLOS style coming out better. Unfortunately, i have no idea of the balance sheet/economics of a journal publisher. Is there any such data that can be compared?

It's not at all clear to me that the end effect is as different as the proponents claim.

Were you born in that ivory tower, or did you just move there before you could read?

Go get a job outside of research for a while (software companies are hiring!) and then take another look at this question. Those of us on the outside can tell you the difference between PLoS and, say, Nature. It's about $200 per year per journal, or $30 per article in small quantities. More importantly, it's the difference between being able to quickly skim an article in thirty seconds with one click and paying to skim each and every one of the hundreds of articles that are required to really get a firm grasp on a field. It's the difference between being able to simply link your most interested blog readers to the original research (complete with original data and figures) and having to tediously paraphrase everything you want to convey, a paraphrasing process that often goes awry.

Your response would have been much stronger without the condescending "ivory tower" and "get a job" jabs.

I was a graduate student for seven years and a postdoc for three. I still count many academics among my friends. I sometimes entertain the thought of once again working for a university. So I'm afraid I'm not very shy about potentially insulting my own social class. ;)

And I'm sorry the subject makes me angry. But it makes me angry! For example, the fact that most of my own published work is trapped behind expensive paywalls makes me very unhappy.

And I use the phrase ivory tower not as a mere gratuitous insult, but because it's the metaphor that seems to fit. Every time this subject comes up, a few people pop up to say that they just can't see the problem: Everyone who goes to school or works in science has one-click access to the whole literature, right? And if I happen to not be a college student or a scientist, can't I just physically travel to the nearest well-funded, publicly accessible university library? Doesn't anyone who matters have such a library at hand, and the time to visit it?

There was a time when printed journals in university libraries were the best we could do. But now it is 2011, and many people no longer read physical books or magazines. In less than a decade the idea of having to leave the house to retrieve written material will be as quaint as having to talk to a human telephone operator to place a long-distance call. In this era, if the attitude I described above isn't "ivory-tower", what is? What should I be reserving the ivory-tower metaphor for?

> It's not at all clear to me that the end effect is as different as the proponents claim.

There have been studies (arXiv used to link to them, maybe still does) that show that open access substantially increases the number of citations a paper receives. That strongly suggests that open access works better for distributing research even within the academy, which is just common sense — open-access papers are easier to discuss, find links to on Reddit and Wikipedia, and often simply obtain. Outside the academy, where it's even harder to measure readership, we can expect the effect to be much more extreme, since most people outside the academy actually have to pay to read non-open-access papers.

Not less selective, less judgemental.

Reply on news.ycombinator.com