For example, say you have a new (patentable) drug that you're trying to get approved and replace an old (generic, cheap) drug. You need to prove that the new drug is at least as safe as the old one and/or more effective than the old one. Which sounds reasonable.
Let's say that the old drug is super safe and that the new one may not be. What you can do is set up your RCT so that you surreptitiously underdose the control arm so the safe drug looks less effective. And then you can say that your new drug is more effective than the old one.
Peer review doesn't notice this because you can easily hide it in the narrative of the methods section. I've seen it a couple of times.
So you can easily have "gold standard RCTs" that get the result you want just by subtle changes in the study design.
Stream or record your experiment. If a 12 year old has the ability to stream his gaming life on twitch, scientists should be able to record what they are doing in more detail.
You could start a new journal with higher level of prestige that only publishes experiments that adhere to more modern methods of proof.
If the answers were easy I'm sure someone would've implemented it already, it turns out those kind of things are hard.
I'd recommend checking out the following article. https://slatestarcodex.com/2014/04/28/the-control-group-is-o...
Critical details and oversights can be lost in translation between reality and LaTeX that could be easily pointed out by third party observers.
And what scientist is going to be happy with constant surveillance that's intended to be shared (so no picking your nose, losing your temper, or making mistakes)?
[0] Literally, thousands. I just finished a paper about a project that we started three years ago. This was the main project for two people, plus a tech: 2000 hrs/yr * 3 * 80% effort = 4800 hours of footage.
The footage is there to support inquiries into how you did your experiment, just as you would skim a youtube video about repairing an appliance to get to the part you need to see.
And yeah, it would have to be a new level of prestige backing this way of documenting your methods. Obviously people prefer being trusted, but p-hacking and replication problems have proven that we can't have nice things.
That is overly cynical. We are already well past that standard.
We require words on a piece of paper that were written by somebody from a recognizable institution, or by an AI operated by somebody from a recognizable institution.
Lots of "interesting" psychology experiments in the days of yore turns out to have lots of damaging confounding variables and we can't just redo the experiment because, well, we shouldn't.
Sometimes this is true. One example is when the people verifying the pre-registration work for a regulator. The introduction of pre-registration seems to be at least partly behind the collapse in pharma productivity:
https://journals.plos.org/plosone/article?id=10.1371/journal...
17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 (χ2=12.2,df= 1, p=0.0005). There has been no change in the proportion of trials that compared treatment to placebo versus active comparator. Industry co-sponsorship was unrelated to the probability of reporting a significant benefit. Pre-registration in clinicaltrials.gov was strongly associated with the trend toward null findings.
But when journals do this the results seem to be worse. They aren't strongly incentivized to improve anything, so you can get studies that claim to be pre-registered but do something different to what the pre-registration said they'd do.