Hacker News Reader

The 90s is also around the time when we pretty much ran out of small molecule blockbuster drugs (the very profitable but low hanging fruit) and started shifting over to much more complex multi-drug therapies like those for HIV and cancer. There are still a few such small molecule candidates left but the major markets like blood pressure, heart disease, arthritis, etc. are very saturated. As the clinic trials start to target more complex diseases with more complicated drugs and therapies it adds a large confounding factor to the decrease in the number of successful trials.

I think you make a very salient point.

From the abstract:

> We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease.

So this is for a narrow field where there are effective drugs already and where all the low hangoing fruits are fully picked because the business opportunity is huge.

> This is a problem related to A Priori Probability, where you're essentially turning research, which could have been statistically applied to a larger population given some statistical confidence, into deductive reasoning which doesn't necessarily apply beyond the finite sample size [1] (also see @entee's comment and links for better explanation of why this is important).

Maybe you are using these terms differently than they are used professionally, but this is not correct if it is meant as a criticism of Bayesian methods.

There is nothing inherently better (and in fact there are several things potentially worse) about looking at the data with inductive reasoning about a potentially larger population with assumed baseline properties.

The problem here is simply p-hacking, whether by accident or intentional. This can be done whether you are performing Bayesian analysis involving a prior distribution, or performing frequentist analysis with assumptions about null hypothesis and subjectively chosen rejection criteria.

If you don't register your model validation procedures ahead of time, then after the fact you can data mine among many (or even all possible) model validation scores and only publish scores that show a positive effect.

It's not about one framework of statistics or the other. It's about statistical rigor for whatever framework. It's about pursuing more than one study. And it's about attempting to use holistic model validation techniques that cover wide ranges of anticipated outcomes, to hopefully drive down the chances that you develop a conclusion from the study based only upon a narrow set of outcomes that could be biased.

I am definitely not using the terms as they are used professionally (and certainly not a criticism of Bayesian methods in any way).

It was an attempt to explain in more lay terms how when p-hacking is performed on some dataset, the "findings" can rarely be applied to larger (or different) datasets due to the arbitrary criteria used to reject the parts of the data that aren't conducive to the selected outcome, since the rejection criteria were sort of custom-fit to that specific dataset.

I agree with everything you said.

Was it possible before this change to run a trial and never publish the outcome? If so, it's also possible that people just kept running trials until they got a positive outcome.

It's still possible now.

> while the graphic used seems to imply a trend, it seems like more data would be needed to conclusively show that trend.

And you can tell that by looking at the graphic...? There's also a reference to a X2-test with p=0.0005 (meaning that there's a 0.05% chance that the trend you see is accidental and would disappear with more data).

I think you're right to be skeptical (correlation is not causation and so on...), but not that too little data is the problem here.

By too little data, I didn't mean that they should have collected more data, I meant that it would be nice to see a) other changes within the drug industry from the 90s and early 2000s that may or may not have contributed plotted on the same graph (not just a single line for this one rule as if that's the only event to affect clinical trials since the 70s), and b) more clinical trial outcome data into the future. More generally, I meant that it's hard to attribute historical events conclusively to specific cause-and-effect, because we can never rewind time in order to change one variable and observer its effects.