Disagree. A significant finding is expected to occur due to chance in direct proportion to the surface area for such possibilities. More studies, more forking paths within those studies, more models, all increase the frequency of spurious findings. So one can do everything perfectly and still get “garbage out”. It is certain.
However, when I said “doesn’t replicate” I didn’t have a single attempt with a 5% chance of failure in mind. I had a field’s aggregate attempt to confirm the result in mind, which would include multiple attempts and checking for statistical bullshit and all that.
Under those conditions the chances are vanishingly small of a whole field getting massively unlucky when trying to replicate a well-done study that theoretically should replicate.
That’s what I had in mind, and I still think it’s right.
—
Rereading what you wrote, a different interpretation of what you said is that the original investigators might have done everything perfectly, and nevertheless found a significant result that was spurious just because that stuff can happen by chance. If that’s what you meant, I don’t understand the disagreement, except maybe semantically. I would call a perfectly done study that shows a spurious result “nonsense,” and I would expect replication attempts to show the result is nonsense, even if the process that generated the nonsense was perfect. Maybe you’re just saying you wouldn’t call a perfectly done study “nonsense,” regardless of the outcome?
One could argue that it's a failure of not sufficiently detailed descriptions of the experiment, but it is how it is in different disciplines. A relevant example for that is a bunch of earlier machine learning research (current methods subjectively seem more robust in this regard) some of which were very difficult to replicate from scratch because it was very finicky and relied on many tiny details that simply can't all fit in the couple pages of a standard paper, but it was definitely not nonsense, because it could be replicated if you knew all the best practices from previous experience. I mean, providing code for a ML paper doesn't change whether the research results there are nonsense or not, but many people might have a very hard time replicating that research without the code.