Odds Are, It's Wrong

I'm somewhat amused to see people complain that "commonly accepted" alphas of .05 are yielding high error rates. Surprise: setting your confidence interval at one-sigma yields a ton of false positives. There's a reason particle physics uses three-sigma as a threshold for further investigation, and five-sigma for discovery.

The difficulty is that in biology, psychology, and other really complicated systems, attaining even two or three sigma can be difficult and expensive. That suggest that we should continue to use low-confidence results... but accept their tentative nature instead of bestowing them with the aura of truth.

And before you start treating a few hundred thousand people with a drug, get a bigger sample size. :)

One thing to note is that there's a ton of selection bias: noteworthy findings get published, non noteworthy ones do not. Therefore scientists will submit papers containing noteworthy findings.

If you do 1,000 experiments in your lab, and (a) absolutely nothing interesting happens; (b) you get a naturally noisy dataset on each, and (c) you write up only those phenomena which would occur randomly only 5% of the time

... you will submit 50 articles, each of them claiming something noteworthy, and all of them false.

Perhaps there are a few actually noteworthy things that were discovered in your lab. ...in which case in addition to the 50 BS papers, you'll also submit 1 or 2 that describe some real effect.

A great article on how the Naymen-Pearson Hypothesis Testing is not particularly useful as a method of inference (e.g. the way most people tend to think of statistics with p-values and 5% significance doesn't say much). Some of what he is saying is a little shoddy, since Fisher actually came up with a much more robust, but less used, method of inference called Maximum likelihood estimation (MLE), which currently stands with The Bayesian Theory as the competing (yet partially complimentary) theories of statistical inference. Strict, but arbitrary, distinction between the null H0 and alternative H1 hypotheses, which is what p-values suggest, is as careless as the article points out.

I see no one commenting on this interesting article has yet mentioned an article on experiment design by a scientist-hacker, Peter Norvig,

http://norvig.com/experiment-design.html

that I mention frequently in comments here on HN because

1) it is accessible to general readers,

2) it is written by someone with excellent formal training and work experience in the subject,

3) it links to excellent primary source literature for further reading,

and

4) it gives cogent, thought-provoking examples relating to errors of reasoning that are common even among trained scientists.

Because I like science, and I like science to progress based on firmly verified factual conclusions, I like my friends here on HN to know about the pitfalls to look out for when anyone does experimental research. Thanks for the submission of the latest journalistic article on related issues here, which also gives us a lot to think about and which has elicited informative comments.

This is an interesting critique, but equating science with "a crapshoot" is journalistically irresponsible. IANAS, but I read a few science blogs, and the impression I get is that most scientific papers hedge and qualify extensively, laboriously mentioning statistical significance, noting possible alternative explanations for the observed data and imagining future experiments that ought to be run to provide extra confidence in the findings.

He gives the example of all the studies linking genetic variants to acute coronary syndrome, only one of which was confirmed in a later (presumably larger?) study. Well, what does it mean for a study to link a gene to a condition? Maybe they concluded, "gene FOO19 causes heart attacks, bitches!" Or maybe it was more like, "4 out of 5 coronary sufferers we studied had this FOO19 gene. Someone should look into it." Those are two very different statements; one is way out-of-line, and the other is responsible reportage.