Hacker News Reader

Comment by pmichaud | Hacker News Reader

This is a really great article, and one thing in particular that struck me was something super obvious in retrospect but that I didn’t think of before:

Replicability doesn’t mean the study was right. If a study doesn’t replicate then it’s almost certainly nonsense. If it does replicate that just means that it’s self consistent—but if it was garbage in, then it will invariably be garbage out.

View on HN

> If a study doesn’t replicate then it’s almost certainly nonsense.

For some fields, especially natural sciences, this statement makes sense. However, for sciences of the artificial (to use Herb Simon's term) I would respectfully disagree.

For example, sometimes the underlying context has changed. I remember a luminary in my field (HCI) once argued that we should consider re-doing many studies every decade, because each cohort uses a different set of tools and has a different set of experiences, and because the underlying demographics of those cohorts change.

This is what we should be paying people like rscho upthread to do. Graduate students should cut their teeth on half a dozen replication studies. Stop chasing novelty at all costs. Instead improve process, methodologies, statistical analysis, etc etc. Our system doesn't pay for that though, so instead we get to live in the world where researchers actively and knowingly publish junk so they can feed their family.

Replicability shows that your method leads to consistent results, not that your hypothesis correctly explains the cause of those results. Yes, your intervention did provoke the causal chain into action, but it may not necessarily have correctly identified or thoroughly characterized the component you identified as the trigger. Your method may work great but only under the perverse set of conditions you happened to explore.

Conversely, if your method fails to drive the desired outcome, your hypothesis could still be correct, just incomplete. Maybe your perturbation of the chemical reaction didn't quite reach the activation energy. Or maybe other essential components in the mechanism were overlooked, given the set of conditions you happened to explore.

Complex black box systems like the brain are notoriously perverse in reproducibly giving up their secrets, even when your hypothesis is correct and your method robust.

“doesn’t replicate then it’s almost certainly nonsense”

Disagree. A significant finding is expected to occur due to chance in direct proportion to the surface area for such possibilities. More studies, more forking paths within those studies, more models, all increase the frequency of spurious findings. So one can do everything perfectly and still get “garbage out”. It is certain.

I’m not super sure I understand your point, but I think you’re saying that it’s possible to run a good replication attempt on a good study and still have it not replicate. I agree with that. I’m not super sure how to correctly estimate the chance of that happening, but one dumb way I can think of is just using p value, so if it was .05 then you have a 1/20 chance of failing to replicate a study even if everything was done correctly.

However, when I said “doesn’t replicate” I didn’t have a single attempt with a 5% chance of failure in mind. I had a field’s aggregate attempt to confirm the result in mind, which would include multiple attempts and checking for statistical bullshit and all that.

Under those conditions the chances are vanishingly small of a whole field getting massively unlucky when trying to replicate a well-done study that theoretically should replicate.

That’s what I had in mind, and I still think it’s right.

—

Rereading what you wrote, a different interpretation of what you said is that the original investigators might have done everything perfectly, and nevertheless found a significant result that was spurious just because that stuff can happen by chance. If that’s what you meant, I don’t understand the disagreement, except maybe semantically. I would call a perfectly done study that shows a spurious result “nonsense,” and I would expect replication attempts to show the result is nonsense, even if the process that generated the nonsense was perfect. Maybe you’re just saying you wouldn’t call a perfectly done study “nonsense,” regardless of the outcome?

There's a bunch of disciplines (live animal experimentation, microorganisms and organic chemistry come to mind, though they aren't "my" fields) where it's genuinely difficult to perform experiments, where it's reasonably common for experimenter to screw up the experiment in multiple ways, and failure to replicate may just as well indicate not a flaw with the original experimenter but a weakness of the skills of the team trying to replicate.

One could argue that it's a failure of not sufficiently detailed descriptions of the experiment, but it is how it is in different disciplines. A relevant example for that is a bunch of earlier machine learning research (current methods subjectively seem more robust in this regard) some of which were very difficult to replicate from scratch because it was very finicky and relied on many tiny details that simply can't all fit in the couple pages of a standard paper, but it was definitely not nonsense, because it could be replicated if you knew all the best practices from previous experience. I mean, providing code for a ML paper doesn't change whether the research results there are nonsense or not, but many people might have a very hard time replicating that research without the code.

Yes, a semantic disagreement: I don’t think of spurious findings as nonsense. Nonsense has negative connotations that I feel contributes to the real problem which is the competitive nature of research and misconceptions about “error.” Too often I hear people elevate the researcher in a sort of “great man” theory of scientific progress. Too rarely do I hear praise for those doing boring replication studies and the like. I worry that younger people hear someone published “nonsense” and think of it as an indication of the quality of the researcher. All of this leads to more p hacking, more avoidance of replication, and more inefficiency in the scientific process. The economic incentives are such that universities compete for big name researchers, those who don’t publish nonsense. That is the problem IMO.

Reply on news.ycombinator.com