Garden of forking paths: Why multiple comparisons can be a problem (2013) [pdf]
stat.columbia.edu
stat.columbia.edu
But for ex post diagnosis, specification curves are also awesome: https://www.nature.com/articles/s41562-020-0912-z
There's also a lot of scope for drama and problems at that second stage, because there can be a lot of wiggle room in whether what was done meets or doesn't meet the pre-registered plan. It might sound easy to follow a pre-written plan but for instance if what you're doing turns out to need adjustment due to discovering something you hadn't anticipated, then technically you might violate your pre-registration. In turn that can yield perverse incentives to not fix things.
Having been through a few rounds of "here's a new procedure that will yield better science, oh no it isn't really working" I'm not really enthused anymore by this kind of tactical fix. From a software dev perspective it's sort of like constantly hot-patching a system in prod to try and cover up bugs when the underlying problem is actually bad architecture that creates too many bugs in the first place. Getting overly fixated on P-hacking or non-replicability distracts from the deeper underlying problem of a mis-specified utility function.
Example: replication was meant to be the big hammer but most studies still aren't being replicated and when they are they sometimes use lax definitions of "replicate" that wasn't what people had in mind, e.g. they'll get an effect size 10% of what was originally reported and still label that study "replicated" or they'll get totally different numbers but claim that because the trend is sort of similar, that's replication.
All that is true, and we should realize that there are no silver bullets anywhere. Cohen/Gigerenzer's painful papers on power were some 40 years apart, yet nothing happened in between. There needs to be a systemic answer, you're completely right about that.
But then, It's important to consider what the assumed model of change is. I believe deeply that influential individual cases can often push progress more than gradual toiling. Having a replication crisis with Daniel Kahneman writing "fucking solve it" emails definitely helps. That's because progress here is a ratchet (new methodological standards set a new lower bound). And more and more people realize that your research today is what you may be judged on tomorrow. So it's a dangerous game to intentionally deceive.
In the end, science needs to be eventually correct in finite time. The steeper the slope, the better. The one thing we must not allow is for the slope to become negative.
For example, there was a big push to give people access to the raw data. Researchers were made to sign data availability statements. The appearance of improvement was created. Then one day someone decided to test if these statements meant anything and discovered they did not; compliance was extremely low [1].
This seems to be how it goes. After over a decade of noise about P-hacking and low replicability in the social sciences, estimated replicability of studies has hardly changed, nor has statistical power improved much [2].
And these are the subtle problems that people don't mind talking about because the problems are sufficiently opaque enough that the general public doesn't pay much attention. During COVID we saw just how deep and serious bad science can get. Climatology isn't better. P-hacking is an irrelevant concern if you aren't even comparing your hypothesis to actual data to begin with (c.f. computational epidemiology), or if your response to a hypothesis/data mismatch is to edit the data (c.f. surface temperature datasets). Poor replicability is likewise irrelevant if the supposedly top researchers in a field can't implement a model that can even replicate its own outputs, and literally nobody anywhere within the institutions cares!
The core problem here just doesn't seem to be something you can fix with this kind of incremental approach on top of the academic basis. You get the temporary appearance of progress in one or two fields where noise is being made, but it's fake and falls apart the moment someone examines it deeply or looks at a slightly different field.
Ultimately, academia doesn't institutionally care if the output of its professors is correct, only that output happens and gets attention. You can't fix this by requiring more forms to be filled out; these people are smart and have all day to find workarounds. For as long as their position and status in society revolves around creating mass/public perception that they're doing intelligent work, vs actual hard-nosed evaluation of said work by independent but deeply interested outsiders (e.g. companies that want to build things), we will continue to be disappointed by the failure of reform efforts.
[1] https://www.jclinepi.com/article/S0895-4356(22)00141-X/pdf
[2] https://fantasticanachronism.com/2020/09/11/whats-wrong-with...