Hacker News Reader

Comment by bluGill | Hacker News Reader

Most science isn't easily reproduced. Often it takes a lot of time and money to get the right equipment. The worst case is super colliders where there is one in the world capable of doing it and so we have to trust the staff. Even simple cases though is still significant effort. There isn't much low hanging you could reproduce it tonight in your basement science left.

View on HN

The funny thing is that supercollider people try way harder to reproduce it whenever possible than social or nutritional sciences group. e.g. Higgs boson detection was only made public when it was found by two independent teams looking at different data(CMS and ATLAS). Also astronomy is good example where reproduction is the norm.

Yeah, particle physics and astronomy is the height of reproducibility, nothing compares to it. The fact that people can get very antsy when observation misses theory just by 10^-9% shows a great scientific environment

Yeah ok but most science doesn't need a supercollider. Most of the erroneous science is in the social sciences, psych especially, and there really isn't any equivalent of a supercollider in psychology. The only possible equivalent is a massive study of like 200+ subjects but once you're at that scale you can be pretty confident that your statistics have converged, anyway. The real issue is low sample size(<100 although most studies done are <30) psychology studies which are by and large relatively easy to reproduce.

I don’t think all the problems that cause lack of reproducibility come from small sample sizes. For example, P-hacking is a thing (intentionally or not) and larger sample sizes don’t solve that. Experiment registration can help so you can track negative results but that doesn’t help if there’s no reproduction attempt (ie you could just have gotten lucky). There’s also straight up fraud you have to deal with.

The point is, op is right that it’s expensive. The computer industry that we’re in claims to be data driven but I’ve observed numerous poor quality studies being done to drive decisions that I’m pretty jaded (no reproduction, poor sample sizes, skewed sample sizes where it’s employees, etc etc). And these are smart people where the decisions being made can impact the financial outcome.

It very much depends on the field. Sure there are some where reproduction is a massive time and resource overhead.

But we don't even seem to be picking up the low hanging fruit - I used to work as an algorithm researcher, and reproduction there (for anyone who set up their experiments logically) is as easy as running a script and waiting x hours for the results to land. Yet reproduction studies were still a novel concept in that field, rather than a standard part of the submission workflow.

Reply on news.ycombinator.com