The most important statistical ideas of the past 50 years
fermatslibrary.com
fermatslibrary.com
(This non-zoomable format with random comments that add no value is extremely annoying... What are people thinking when they do this.)
Being able to do multiple analysis of the same data, and find one where p < 0.05 has revolutionized academia.
Entire careers have been built around p-hacking.
Of all the statistical idea, it is the one which arguably has had the most influence in the life of the academician.
It's much much harder to lie with effect sizes. For instance, in genetics there are entire careers built on an association of certain genetic polymorphism with a particular phenotype. Even if the p-value can be reproduced with huge sample sizes, the effect size of the association is usually tiny.
In other words, if you carry the risk allele the odds of developing a disease are usually only slightly bigger than if you didn't. But this is often not told to funders! Focusing on effect sizes would drive research effort to things that actually make a difference.
You can do effect sizes with something as simple as a confidence interval, or with a fully fledged Bayesian model that tries to estimate the posterior distribution of some parameter.
This really should be the standard for any meaningful science.
That said, you can do something called multiple comparisons corrections, where you take into account the fact that you're doing multiple tests in the same go. In this case you'll end up being more conservative, and it's not the false positive rate you'll be controlling - instead you'll control the family-wise error rate, which basically has to do with whether _any_ of your positive results are false positives.
Submiting a large number of individual registrations for a single set of data is just not the norm and so it would be suspicious to begin with. (Okay, I will admit that on the order of 2-3 registrations for very different categories of study may not be as strange; but certainly 20 would signal something very odd).
While impractical in many cases, I think the gold standard is to approach a problem from as many angles as possible, and relate it to disparate sources of knowledge, including theory.
For example, the accepted values of the physical constants are established by munging the results from a wide variety of different kinds of experiments, guided by theory. They don't just repeat the same experiment over and over. These constants are robust enough that you can hang your hat on them for virtually any practical use.
Yeah, the term only came up recently. But ultimately p-hacking is "just" exploiting freedom in the possible interpretation of statistical results. Also it wasn't really "invented", it's just what humans do if they have data in front of them and some expectation what the data should tell them. I'm pretty sure the first p-hacker was the first person to do any form of early statistics.
I guess the greatest invention would be a method to prevent p-hacking. (And yeah, I know about preregistration. It doesn't work the way it's done right now, because a) most people don't do it, b) preregistration is rarely precise enough to really prevent p-hacking and c) nobody cares if you switch your outcomes.)
It is the "Stationary Bootstrap" because it can only be used on stationary data (for very obvious reasons). Problem is that tests for stationarity are very poor.
I have seen it used in academic finance to prove technical analysis. Yes, really (A clue to the clueless - except in very special circumstances technical analysis does not work). Completely inappropriate to bootstrap financial returns data as they are not stationary.
I am talking about White's Reality Check: which was devised by Herbert White who was a Distinguished Professor of Economics at the University of California. It is completely bananas, (final return data is not stationary. Bootstrapping it gives random noise. Duh!). When I looked at it about decade ago the paper it was developed in (two papers actually) were among the most cited in the field.
I am not convinced that it can be used in stationary data, though I only have experience debunking it, not making good use of it. If something seems too good to be true.... And the bootstrap promises to take your too small number of data and make it into a lot of data.
I see the paper here did not bother with stationarity. I stopped reading there. I cannot bear the myths and self deception statisticians tell themselves. I did it as a graduate student for far too long.
And the bootstrap is not for taking a small amount of data and making it a large amount, it’s an empirical estimate of a sampling distribution.
More importantly, they're people and probably had _some_ reason for doing what they did. If we'd prefer an HTML version instead or in addition, we ought to make an honest effort to understand their perspective and address their concerns rather than just bemoan their choices.