Hacker News Reader

Multiple comparisons? 1 million independent tests? Hello?

>Your P value is 0.0002. This doesn't count as discovery, but clinically I know what my judgment is going to be.

>Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at.

^Means you should start a new trial with n > 20 given the same placebo/drug split.

In the scientific world, it is economically infeasable (cost/time) for 1 million tests for a given hypothesis. Even n > 20 can be difficult for certain studies. Bootstrapping the results to simulate 1 million trials won't fix the aforementioned issue either.

> Even n > 20 can be difficult for certain studies.

For some expensive experiments even n=3 is difficult to reach. "Should we do another replicate and go from n=2 to n=3, or should we hire another post-doc?" is a relatively common (rhetorical) question.

Reply on news.ycombinator.com