Scientists lift the lid on reproducibility (2016)
nature.com
nature.com
* a clinical trial had (insufficient sample size|no controls|other methodological error) and is excluded from meta-analysis for this reason
* engineering analyses that turn out wrong (an early paper that postulated you could use a lightly modified jumbo jet to seed the stratosphere with sulfur or early analysis of LWR accidents)
* numerous theoretical physics papers that are a shot in the dark (e.g. papers about superstring theory if the Universe is not supersymmetric, all the competing models if it is)
* math-heavy papers with 50 pages of calculations and a sign reversed on page 23
* CS papers based on software that was wrong (I looked at the source code and fixed the bug)
* one I wrote (nobody ever told a group of physicists how to tell if a power law is a good fit to a probability distribution or how to estimate the slope)
When I have been out of the academic loop and had to spend $30 to buy a copy of an academic paper that we thought would help in a commercial project it has often been a poor investment because the content wasn't useful.
Of course some scientific papers are highly valuable but I'd say one reason I never found my voice writing scientific papers was that I was aiming for a level of quality that was unrealistic and not representative of what I was surrounded with.
Also why aren't the former type of issues caught before being published in the paper?
A peer reviewer is not going to go through complex calculations line-by-line or find bugs in the software. The person who does find that is a person following in the footsteps of the researcher who wants to build on that result.
Software bugs are common in commercial software too but if they are put in front of customers most get fixed.
What boggles my mind are the poor standards in biomedical work. Math-heavy papers are often written by one or two people and a computer but clinical studies involve a lot of people, paper and you'd think formalized process.
I remember one paper where they tested COVID-19 patients and could not detect Vitamin C in their blood but didn't test anyone who didn't have COVID-19. Vitamin C levels vary greatly in different populations and the test is tricky and requires careful sample handling... It could be they were unable to detect Vitamin C at all, but they only way they could disprove that would be to use proper controls.
Most clinical trials are rejected when it comes time for a meta-analysis, I think that journals shouldn't be accepting studies that wouldn't be accepted for meta-analysis and work that isn't to that standard shouldn't be funded.
My brother has a PhD in biochemistry and quit one job at a lab when he pointed out to the head scientist, a woman who had made her reputation out of a series of related papers and work, of a serious flaw in the experiments that could invalidate the result. Not only did she ignore him repeatedly without acknowledging the flaws as important (note: if my brother was right, the core result was invalidated!) but continued to churn out papers on the same topic.
This comment is not anti-science, mind you. It's just to show that when egos and reputations are involved, all bets are off. Nobody wants to be told they built a house of cards. I'm confident the conclusions from these papers are either irrelevant in the grand scheme of things (so it doesn't matter if they are wrong), or would get eventually rejected when other scientists take a look at them.
I find it funny that you feel the need to say this, but I also 100% get it. In my experience, there's no one more skeptical of scientific publications than people actively doing research, but it's such a delicate balance to do so without upsetting the generically pro-science laymen. It's like a modern day shibboleth, come to think of it.
Back when I was in academia, I had a similar experience. Someone gave a talk and I immediately had doubts about the method used. It was known in a related area that methods like the one presented tended to appear more accurate than they actually are. So I spoke with the researcher privately after the talk. I didn't expect this researcher to know about the problem, and I think I said that it's possible everything is okay but some extra checks need to be done. They seemed interested then, so I sent them an email with some papers on the problem. I never got a response, and their paper was published without even mentioning what I said. The paper now has over 100 citations and that research group has built on the method I had a problem with.
In my experience the people who focus on high quality scientific output are some of the smartest and most successful people in the field. The challenge is that you have to be productive as well. This is why we don't have error-free science, just as we don't have bug-free software. (And it should be said: unless you produce bug-free software [or whatever it is you do professionally] there is probably someone who could cherry-pick your errors and present the same case about your work. It's important to approach life with humility.)
Also in part because papers are “dressed up.” That’s a polite way of saying that the description of what they’ve built has been stretched. Authors need to make reviewers excited. They do this with ideas, not code. The purpose of their code is to show that the ideas could be practical, not that they are. So research software is almost always incomplete.
Mostly kidding.
Science and published knowledge as of today looks mighty fragile, given it's built on empiric data from these experiments.
This seems like an underrated part of it. The meaning of 70% is a bit unclear to me since the article didn't cite how many replication studies most have done. If most have done 10 replication studies and weren't able to replicate a component of it they would count towards the 70%.
Given the quality of research in the social sciences I would suspect the issue is much more significant than that - but nonetheless a distorting way to report the stats IMO.
It is important to remember that the "median" paper is cited less than 5 times, which means that there may be a large fraction of results that cannot be reproduced, but also have had little effect on science. There have also been highly cited papers that could not be reproduced but whose results were correct (break through experiments can be hard to reproduce), and there are highly cited papers that were simply wrong. But it is hard to measure these phenomena in an un-biased way.
The exact reproducibility of a paper is certainly important, but science is much more about building on results than reproducing experiments.
Has the rate of reproducibility change at all over the decades?
Are we just simply more aware of the problems today than in the past?
When you had 10 papers published out of which 7 were BS, you could weed those out by reading each.
When you have 1K papers and 700 are BS ... you are not going to be able to do so.
Instead of constantly prioritizing the erudite publishing system (whose goal is conceptual in nature), I think a bit of openness from a utilitarian perspective can seriously help scientists with troubleshooting and reproducibility. The proof of the pudding is in the eating.
Some previous discussion from 3 years ago:
https://news.ycombinator.com/newsguidelines.html
(Submitted title was "70% of researchers have failed to reproduce another scientist's experiments". We've changed it now.)