Flipping 12 coins and having them all land on heads is a 0.02% probability event. Cancer spontaneously going into remission in a patient is pretty far away from a coinflip.
Maybe I should have put this in a more productive way: I prefer studies that have a large enough n that you can actually measure the failure rate. There's a classic statistics paper where an operation was done on 3 patients and none died- the paper's title is "If nobody dies, is everything OK?" and goes into whether you can reliably conclude causal effects with very small n.
Further- let's say the drug caused remission in 50% of the patients, and the other 50% had spontaneous remission (or remissionc aused by other aspects of the treatment). Now it's not 1/2 * 12, it's 1/2 * 6 (or whatever base probability you want to choose). If you compare that probability to the number of people in clinical trials in the US, you would expect that after 30-40 years, you'd see at least one trial with 12 patients with complete remission.
(I don't actually doubt the drug is effective. We already know that. And we already know that studies this small really stretch the concepts of significance and causality).
What's really cool is that papers tend to have sections where they discuss the statistics so you can use real numbers. Their null hypothesis was a 25% response rate.
"This decision rule would result in a type I error rate of 6% if 25% of the patients had an overall response and would provide the study with 84% power if 50% had an overall response. The null hypothesis was established on the basis of a study by Seligmann et al., in which the observed response to chemotherapy among patients with mismatch repair–deficient rectal cancers was 7% (8 of 115 patients)."
There's also publication and hacker News front page bias to consider in the odds making as well...
You don't actually have to run that many trials before a 0.02% liklihood event happens by chance.
Considering how many of these get conducted and the odds are often better than 50/50, I expect multiple trials have cured everyone.
If colorectal cancers in 2nd or 3rd stage actually had a fifty-fifty chance of spontaneous remission in the real world, the disease would be a lot less scary than it is.
They don't. Spontaneous remission does happen, but isn't common. The chance that a 12-patient trial produces spontaneous remission in all 12 patients in a truly random fashion, is much smaller than 1:5000.
And the GP actually mentioned it. But everyone reacting to him still addresses the 1:5000 value. It is a nice example of anchoring as described by Daniel Kahneman - here is a number, let's cling to it.
THat's how it works in clinical trials. I didn't make my statement out of ignorance- I worked adjacent to trials for decades and have seen all sorts of claims of results that didn't hold up to scrutiny.