Hacker News Reader

Comment by mxcrossb | Hacker News Reader

> despite not actually trying to replicate the Fleischmann and Pons experimental setup accurately

I’ve been seeing this claim in another field: about hydroxychloquinine. Every time a study comes out which doesn’t show a benefit, people rush in saying “that’s because they forgot X, didn’t dose the right amount of Y, etc”

I’m actually not sure what to make of this phenomenon in general. I will say though that when you’re trying to test a hypothesis, completely replicating another experiment isn’t necessary. You should test it under a variety of conditions. That makes it easier to avoid the same hidden mistakes that the original experiment might have included.

View on HN

You should definitely also try the exact original conditions. Think about reducing a tricky bug: you start with the exact input and code that first caused the problem, then try removing constraints to see what preserves the bug.

Ed; I should clarify: you start with whatever big nasty test scenario you need to make sure you can reproduce the effect at all, and only then can you start changing the conditions and be confident you're still measuring the same thing.

Yes but science and computer programming are fundamentally different activities. The programmer tries to reproduce the bug because his main fear is that he can’t see its effect. But the scientist’s main fear is that the experimental setup does not robustly test the underlying hypothesis. If you reproduce exactly the conditions (and remember you will always be criticized for missing X), the best you can learn is whether the previous result was not completely fabricated (or statistical noise). It’s much more productive to subject a result to a wide battery of tests.

If you're dealing with an unknown, confusing phenomenon, you don't know what the underlying hypothesis is. You don't know yet what constitutes rigorous testing and what is just wandering off into the weeds. So you need to start with seeing if the thing happens at all. You do in fact need to know whether the previous result was garbage before it can be productive to do a wide battery of tests. And "wide" depends on how sensitive to conditions your effect is, which you also don't know yet. Your closest thing to an anchor is the initial claimed setup, so start there.

Reply on news.ycombinator.com