5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at.
This would still end up leading to misuse of P-values. Let's say you're doing a genome-wide association study on several hundred thousand SNPs. The traditional threshold is 5e-8 (0.05 / 1,000,000 effective tests). So using 3e-7 for the threshold for "discovery", you'd count many things as discovery that shouldn't be so.
On the other hand, let's say you do a study with 20 people with cancer. You give 10 of them a drug, the other 10 a placebo. All 10 with the drug survive; all 10 with the placebo die. Your P value is 0.0002. This doesn't count as discovery, but clinically I know what my judgment is going to be.
This is all to say that the misuse of P-values does not just come from the threshold.
A friend of mine is researching cancer threatment at a commercial lab. They make a lot of long shots. Lot's of PHD students are hired, and they let them work on 1 of those longshots.
Costs them milions of euro's, but potentially gives them bilions.
These studies survive on statistics. It is even tweaked by having cross validations where they leave out the worst bins, 'because they are extremes'.
Always be very suspisious about statistical proofs, especially from companies with great commercial interests.
>Your P value is 0.0002. This doesn't count as discovery, but clinically I know what my judgment is going to be.
>Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at.
^Means you should start a new trial with n > 20 given the same placebo/drug split.
For some expensive experiments even n=3 is difficult to reach. "Should we do another replicate and go from n=2 to n=3, or should we hire another post-doc?" is a relatively common (rhetorical) question.
This is an enormously common problem even with current p value thresholds. It's part of the reason why you see dramatic "A causes B!" results followed by replications saying "well, only a little."
http://www.statisticsdonewrong.com/regression.html#truth-inf...
It's a bad idea to use p-values as thresholds, regardless of where you put the threshold.