Comment by arcanus | Hacker News Reader

Comment by arcanus | Hacker News Reader

"Indeed it has been said that Democracy is the worst form of Government except for all those other forms that have been tried from time to time."

-Winston Churchill

-----------

While I broadly agree that the dissemination of academic research can and should improve, it is not clear to me what will amount to a superior (while still robust) system. Nor does the article clearly articulate a viable alternative.

The mathematical sciences have provided remarkable utility for the last 350 years in part because of the robust feedback loop of, 'hypothesis-experiment-disseminate'. I believe this, 'science is broken' hype is largely just that. No one throws out a codebase before a superior (and regression tested!) alternative is ready to be deployed.

My 2 cents: 1) Let's push for publicly funded research to be made publicly available, regardless of what journal it was published in.

2) Let's think about incremental changes to peer-review that can improve dialogue and speed-up review times. Instead of waiting 6 months for a late paper reviewer to drop a series of comments in my lap, why not a wiki or github-like editing and annotating environment where we can (anonymously) generate comments and a fruitful dialectic. Then, let's include the comments (again, anonymously) with the paper. Show how the sausage is made-- scientists have little to hide.

FWIW; I am a practicing scientist in the field of computational engineering.

It feels like we've lumped together a bunch of different use cases and metrics into the current publication system.

1) Authors need to get peer review, feedback to make their case stronger and improve the communication.

2) Researchers need a way of filtering the publications so they can find the gems in any field or ones very useful to their own work.

3) Then there's the need to evaluate how well particular people are doing in their research, which boils down to metrics including the prestige of the journal (immediate) and the number of citations it'll eventually accumulate (long term). I'm not aware of any distinction between citations of a work in an introduction or review and citations of a work that provided some essential part of the current method. These are pretty big differences since the former is more strongly influenced by reputation while the latter is its utility.

Here's how one alternative system would work:

Start with a github-like editing and annotation system for soliciting and coordinating feedback. This also allows readers and possible evaluation systems to see the types of contributions people make.

References would get more sophisticated with categories like "background material", "thought provoking previous work", and "indispensable to this work."

Publication initially goes to general repositories like arXiv where anyone can read it. Have an amped-up Google Scholar search where you can easily bookmark works of interest, tag them, and share them with colleagues. Different HN or reddit systems would spring up to bring better work to the attention of more researchers.

"Journals" have a panel of editors who select a subset of work to display under their banner. This is not mutually exclusive so particularly awesome work would be featured in Nature, Science, and more specialized journals in this alternate universe.

There would be a diversity of impact metrics including ones based purely on "my work would be impossible if not for this work X". Affiliation between authors could be factored into some metrics so greater impact is given to referenced work with more degrees of separation from the author's current and previous institutions.

I like the concept of semantic citations, e.g. "cited for methodX", but it's hard to make a general enough system out of this that doesn't disrupt the flow while reading. And in a way, we already do it, except that it's not being done in a consistent fashion that enables datamining of it. Getting publishers to standardize some markup for this would be cool, but I don't think the outcome justifies the added effort for everyone.

As for several journals publishing the same thing: there are many problems withh that idea, not the least that Nature/Science doesn't want your paper with an 11 page methods section, they want something much closer to a popsci article. So you have to write several. This is already happening today, when someone gets to publish their supercool finding in Nature they also write one or more longer, detailed (i.e. proper) papers that are sent to more specific journals.

"I like the concept of semantic citations, e.g. "cited for methodX", but it's hard to make a general enough system out of this that doesn't disrupt the flow while reading."

The different types of citations would be different markup tags and the display to the reader wouldn't be any different than the current reference notation.

Regarding the difference in ideal paper format between journals, I think there are already multiple detail levels prepared. Our Nature paper, for example, has a pretty significant supplemental information section. The standard could be three levels of detail: overview, standard, and detailed.

If the reader can't see the semantic markup, then what's the main purpose, and how does this purpose justify the additional effort required, both to implement this system in a standardized way across journals, and by authors who have to tag their citations with this semantic markup?

It's for the algorithms that process papers to produce metrics. Display systems can choose to ignore it and show them all as [1] or might use color, hover message, or whatever to show the distinction in a way that doesn't alter the standard reading experience if that's desired.

Then I think there's no way that justifies the additional effort required both from people writing papers and for journals to implement this.

A reasonable compromise IMO is for Journals to have a very temporary lock on new publications. Public access after say 1 to 2 months seems to meet the vast majority of "Open Science" requirements without much change required.

2) IMO a much more stringent form of peer review is necessary. P<0.05 is only useful in very limited situations. I would argue that unless you republish your exact experimental design beforehand you would need to drop that to P<.01 to cover for a lot of marginal research out there. The old do 10 studies and you can likely publish one of them is harder when you need to do 50 or fudge the numbers harder.

> 2) IMO a much more stringent form of peer review is necessary.

That disregards the cost of review. Reviewing and repeating experiments uses resources that could be spent elsewhere. On the long tail of papers that are not high impact even if correct, that's probably not a good allocation of effort.

What I think would be better is that as a paper crawls up the impact curve (probably measured by citations), then additional resources should be spent validating and repeating it.

In theory, that's how the system already works since everyone wants to build on top of or refute famous papers. In practice, I don't know if it does.

I am not suggesting people need to necessarily spend more time doing the review, just accept less BS.

In many cases the bar is far to low.

Repeating experiments is not a wasted effort. "More than half of biomedical findings cannot be reproduced – we urgently need a way to ensure that discoveries are properly checked" - Elizabeth Iorns is co-founder and CEO of Science Exchange.

Science which can't be repeated, that is problematic.

[1] http://www.newscientist.com/article/mg21528826.000-is-medica...

The peer-review before an article is published is just a sanity check, like Fizz-Buzz during an interview.

The more valuable check is what happens after a paper is published, when your peers read the article and discuss it and critique it and try to replicate it and publish responses to it. What makes peer-reviewed journals valuable to science is the that lots of practitioners read it; the value of peer-review is that it makes the journals more interesting for practitioners to read because it doesn't waste their time on obvious crap.

Publishing a paper is not a stamp of truthiness. It's a communication to the community that something is interesting. Maybe it's right, maybe it's wrong, time will tell. Truth emerges from a large body of research and publications from a large number of researchers. Any of the individual papers along the way may be missteps; it is only in hindsight that a particular paper can be picked out as seminal or definitive.

That's OK.

It's also a stamp of relevance. Like the front page of HN vs new.

I'm not sure P<.01 helps overall, though I wouldn't cry if it happened.

1) small p-values require larger n's which will price some real studies out of happening, 2) it only makes dishonest research marginally harder and a little more obvious, 3) a stricter p-value may increase the misconceptions about how important they are, we already have enough B.S. flying around because "P<0.05, it's scientific now!".

I haven't totally taken a side but I find the idea of dropping p-values altogether to be interesting as well: https://whyevolutionistrue.wordpress.com/2015/03/05/psycholo...

1 - Publicly available archival-quality publishing for all science done in the US would cost at least millions a year to administer. The idea is nice, but is just an idea without the funding to make it happen.

2 - Peer review problems could probably be addressed on a per-venue basis. Eg in many areas of CS the max turnaround time is a few months. And there's nothing stopping you from publishing peer reviewer's comments. But I am not convinced that enforcing this universally is a good idea, for either the author or the reviewer.

> Publicly available archival-quality publishing for all science done in the US would cost at least millions a year to administer. The idea is nice, but is just an idea without the funding to make it happen.

This is what government is for and a million a year is a rounding error to the government.

Agreed. But I think this is important to point out. It's not as if scientists are lazy or greedy -- they spend the most time and get no benefit out of publications in journals in the status quo. There just aren't compelling alternatives to the traditional model in many disciplines, and most of the time it's due to want of desire rather than want of money.

Someone down voted this? Archival-grade hosting is pretty expensive -- it means the cost of publishing a paper is the cost of serving that paper (and keeping backups, maintaining the software infrastructure, etc) for... ever. Typically both for- and non-profits place this cost in the three-to-four figures range. It's not as simple as "slap it up on arxiv" -- arxiv would break in a second if every scientist started using it as the de facto platform for every paper.

Reply on news.ycombinator.com