Hacker News Reader

Comment by chmod600 | Hacker News Reader

Proposal: if a paper isn't reproduced in ten years after publication, then it gets automatically retracted (which can be reversed as soon as it is reproduced). Any papers that cite the retracted study (in a way that the conclusions depend on it) would also get retracted. That would be powerful incentive for all the researchers who cite the study to try and reproduce it so their papers don't get retracted.

You could still search these retracted studies when doing research, of course. You just can't cite them.

View on HN

All that does it create an industry for replication labs, and then a market where these labs compete to develop the most "guaranteed" replications for the least cost, and then a whole lot of replications which check the box but actually rely on strained interpretations, questionable modifications to process, uncaught fraud, etc.

That sounds worse than what we have since it just eviscerates the significance of what replication means in the first place.

It's almost like the free market is not the silver bullet algorithm many seems to think it is.

Free market? It's government funding doled out by centralized committees that drove this.

It was actually the incentives created by the free market that finally unravelled this whole thing, as per the article.

What part about the described system is free or a market?

This would work. For-profit replication labs could be part of it, but the market would have to be regulated such that:

- any university receiving public funding must participate in mandatory replication

- some % of any grant would be reserved for replication

- replication labs would be licensed/accredited, subject to government inspections

- labs would not be able to choose who replicates their work

- replication labs would be rewarded for the work, not for the outcome

The amount of money for an NIH R01 non-modular budget hasn't changed since 1999. Which means due to inflation, the average "gold standard" grant budget covers half as much as it used to, roughly.

Wherein are you going to fund mandatory replication studies - some of which are massive cohort studies, so you're going to need a whole new population cohort - let alone the...citation police...to review every citation not just for its existence, but for its content.

We just need to cement the notion that no theory is truly proven or beyond attempts to disprove it through replication. Attempts to discourage replication should be a red flag.

The same issues exist. The next dodgy scientist on the hamster wheel looking to get a name for themselves will claim reproducibility and then publish a follow up in the spirit of publish or perish. You could further entrench the issue with this approach, unfortunately.

Better solution might be for the government to just fund reproducibility studies, and even departments of reproducibility. Take the profit motive out of it, find good scientists who are painstaking but maybe not innovative and fund them to reproduce major results. The scientists would never get the credit for major breakthroughs, but could occasionally be wrecking balls that called research like this into question. With consistent and reliable funding from the government their reward could be stability of employment rather than innovative fame (of course these days some partisan politics would probably gut it, which is why we can't have nice things).

I'm open to that idea but not sure how it would play out. It could go wrong in a few ways.

One potential problem is that it could become "the" authority on reproducing results. If they repro something and another scientist can't, or vice versa, would the other scientist get ignored?

Another problem is that it could take away the independence of science. The government might start saying what is good science and what isn't.

They can only publish their results, though, they can't force people to accept them.

Any other institution, anywhere else in the world, is free to publish results which disagree.

Scientists should then have the necessary tools to figure out which one is likely to be correct, and try an independent third or fourth time.

It would be expected that sometimes there would be a failure to replicate which was a mistake in how it was replicated, that shouldn't be seen as being a failure of the goal of replication, and the original authors should be incentivized to reach out and discuss the issues with the methods.

What percentage of research is ever replicated? Are there a lot of scientists doing replication studies?

Is there a level of collaboration among so many people that at least data can be re-used without being "reproduced"? I.e. the field as a whole has put enough work into an apparatus or infrastructure that we can regard the initial observation to have been trustworthy? We shouldn't have to build a second LHC before we believe any claims from the first one, right?

Actually the LHC does have two detectors with completely independent research teams, often doing the same experiment.

Hypothesis comes before data collection in the scientific method.

You can share equipment, but if you share data, then it's the same experiment not a new one reproducing the first.

Sharing data would not only miss fraud, it would miss experimental errors (e.g. spilled beer in the petri dish), p-hacking, etc.

How about the threshold for publishability is that two independent groups come up with the same conclusion? So you have to partner with another team somewhere else in the world in order to get published, and then you get joint credit.

I think there's a powerful incentive to be first, which drives people to research in the face of so much doubt. Trying to convince one lab that the idea is worth funding is hard enough. Two labs is just unrealistic.

My idea hacks around the problem by not diminishing that huge incentive to be first. Even if your paper is retracted due to lack of repro study, you were still the first, and if it does repro you are back to full credit.

I just add an extra incentive to get people who cite the study to verify that its reproducible. And at the same time, give the original author more incentive to include lots of details in their papers so it's more likely to be reproduced, and lock in their fame.

This is a powerful idea. Most problems in science are incentives problems. Sadly there's little or no incentive for journals to adopt this policy.

I agree.

Correcting the many incentive problems in modern American science would need a hypothetical body with significant funding leverage over journals & scientists to exert executive action. Sadly there is no such centralized funding body, so the problem must be unsolvable.

Reply on news.ycombinator.com