Manipulating the Alpha Level Cannot Cure Significance Testing

frontiersin.org

5 points·gattilorenz··2 comments

We rely on science as presented by the media to inform us about everything from what we should eat to how our surroundings influence our behaviour. It is a pervasive cargo cult - no article is considered credible without vague cherry picked references. The whole pop-science/self-help book industry must be worth billions.

Yet we hear about p-hacking and replication crises and seem to adopt an attitude of sweeping inconveniences under the rug.

I don't know enough stats to fully understand the article. Can anyone explain why we should not consider the claims of a huge body of scientific work that have relied on these statistical methods for over 50 years to be invalid?

NHST was always intended to provide evidence within a corpus of replicated research. institutionalized careerism creates disincentives for science to work that way in practice.

it's actually consistent with the NHST paradigm to treat any given result with skepticism; the degree of skepticism depends on many things, some of which are statistical and some of which are domain-specific. but skepticism is not the same as assuming invalidity. imperfect experiments are still (expensive) evidence for something.

imo, what you should reject entirely are media reports (usually based on somebody's PR team) that try to tell you to do anything based on one study or the work of a single lab.

Reply on news.ycombinator.com