Hacker News Reader

As a physicist, I think how we normally describe what you’re talking about is that there must be a robust and explanatory theory that goes along with experimental measurements. The most fruitful areas of research are the ones where theory and experiment exist on approximately the same level of what they can predict or measure. These strong explanatory theories are the things that generate truly robust, reproducible experiments.

In cases where there is significant differences in the maturity between theory and experiment, you either end up with theorists playing games with math, making up tons of unconstrained theories (string theory, for instance), or in the other direction you end up with experimentalists measuring anything and everything they can imagine, half of which is probably not theoretically enlightening, waiting for theorists to constrain their space of possible experiments to potentially fruitful paths.

Obviously this is harder for fields that don’t have models as robust as we do in physics, but I’d guess the same phenomenon happens everywhere.

I'm a physicist too. A bad analogy is that theory and experiment are like plywood. The individual layers are weak, but criss crossing them against one another and gluing them together makes something that's quite strong.

I actually quite like that analogy - I may use that the next time I have a chance to teach.

Yeah, why is it a "bad analogy"? I think there's something useful in it, possibly even essential.

Clearly, for there to be chronic scientific progress, there must be some value in experimental practice. And yet widespread irreproducibility of results negates that value, by definition.

So, where is the progress coming from?

Your argument that "there must be a robust and explanatory theory that goes along with experimental results" helps make sense of TFA, which otherwise presents as a paradox, or a ruse to excuse shoddy practices.

The presence of a viable theory is a useful heuristic for sorting useful experimental practice from less useful. At least, I imagine it is. How can we know for sure?

Has anyone published a meta analysis that checks for this?

And has that meta analysis been independently reproduced?

Of course reproducibility isn't necessary. There's a whole subset of sciences (often called the historical sciences) where reproduction is generally impossible because of the arrow of time. You can't replicate evolving humans again, as an example.

Nonetheless, reproducibility is a really nice property. You have to be incredibly careful with your theory if it can't be justified.

That subset of "sciences" aren't sciences by definition if they are not reproducible. They may be useful but they are not sciences.

If your personal definition of sciences doesn't include evolutionary biology, geology, or large parts of astronomy, it's a very unusual definition. All I can tell you is that most people would disagree.

That's ok with me. I think it's very important to distinguish between the two endeavors and ideally have different names for them so we don't get led into these kinds of otherwise unnecessary discussions. The domains you mention seem (to me) generally worthwhile. But it's very hard to articulate exactly why they seem that way given we can't reproduce them which makes their results hard to justify believing in.

Wait, what electromagnetic results are not reproducable!?

I think the parent poster is saying some EM experiments weren't initially replicated because of the peculiarities of the set up, etc. Another example is that some people couldn't see the Galilean moons initially. This made initial report of them somewhat questionable, as some observed them and some didn't.

> Scientific results can be strengthened by replication, but that's just one thing, and if that's all you do, then you end up with a science that does nothing but generate independent factoids.

Scientific results are what makes science universal; since they can be reproduced everywhere by anyone.

Indeed, at least until they can't be. The whole "reproducibility" business is because of an emerging "replication crisis," where it has been found that a large swath of published studies fail attempts at replication. Articles about this situation usually say something to the effect of "even in physics," though usually without evidence.

A variety of causes have been proposed, such as abuse / ignorance of statistics, bad experimental design, and even outright chicanery.

> In my view, reproducibility is neither strictly necessary nor sufficient for scientific progress.

In your view? Reproducibility is essential to scientific progress because it defines what science is. It's the backbone of science. Hypothesis,testing/experiments. It's what separates science from math, arts and pseudosciences like social "sciences".

> I'm sure if you went back through the literature on something like electromagnetism, you would find results that fail to replicate, yet the theory of electromagnetism if applied properly is remarkably robust.

How would it be "robust" if it couldn't be replicated? How would the theory of electromagnetism be robust if it failed to produce replicable results?

> Scientific results can be strengthened by replication

No. It can only be a "scientific result" if it can be replicated/tested. It's definitional.

> Discovery of a psychological "effect" is perfectly scientific and interesting

No it isn't if it can't be test/replicated/reproduced. It's not science. It's something else altogether.

> And if those sciences are also plagued by irreproducibility, then they may embrace scientific methodology without producing a useful scientific knowledge base.

There is no "scientific methodology" without testability/replication/reproducibility.

All science is based on hypothesis-testing.

I get what you're saying, but even within the more reputable sciences, the body of known or published results is peppered with results that will fail to replicate. They have not all been identified. There will be more. The good results are mixed with junk results yet somehow science progresses. The robustness of science has to come from somewhere other than the complete absence of junk results. This is what I meant by my first sentence.

I read that in the century after Newton, the French Academy offered a prize for evidence of the failure of Newton's laws. They gave out the prize dozens of times, yet Newtonian physics kept getting stronger and stronger. Eventually they stopped giving out the prize. Many of the contradictions apparently had to do with the lunar orbit, which was poorly understood.

Check this article out https://aeon.co/essays/how-popperian-falsification-enabled-t...

Deals with some of your points.