It's solid probabilistic reasoning about the dominant statistical methodology, and its conclusions have been empirically demonstrated in multiple fields.
It's solid probabilistic reasoning about the dominant statistical methodology, and its conclusions have been empirically demonstrated in multiple fields.
- A publication bias term u that is pulled out of thin air for different scenarios.
- A parameter R that reflects the unknowable proportion of true relationships/hypotheses in a field compared to the universe of possible hypotheses. The pre-study odds he calculates are determined entirely by R.
Actually this paper provides a good framework for determining what factors will affect the PPV of a field. In that regard it was a good contribution. But the enormous leap to the title -- which can only be achieved by massive assumptions about R and u -- was baseless fearmongering and demagoguery.
If one wants to address these issues at the 50,000 feet level, it would be far better to look at something concrete, such as reproducibility rates, as indeed many have done, rather than models based on pre-study odds. If we actually knew the pre-study odds, or IOW the proportion of hypotheses in hypothesis space that are true, then we could just correct our p-values for that and be done with it. Although I suppose that many Bayesians would not see any problem with abstracting away everything we don't know into priors R and u, making wild guesses about them, and drawing conclusions.
We do have a replication crisis on our hands. But the pre-study odds, whatever they are, are unchangeable. If this paper had been framed in terms of "the lower the pre-study odds, the higher the power will need to be to compensate to get an acceptable level of reproducibility", I would accept it wholeheartedly, although then it would have been a simple and obvious statement rather than a citation-grabber.
Until such time as someone does bring theory to biology, for example, wet-labbers (not me) struggle on. I am in a little doubt as to whether general theories about complex systems could provide useful predictive frameworks, but if so, great. But until then, pre-study odds are zero if you don't do the study.