Hacker News Reader

Comment by btilly | Hacker News Reader

This whole field seems likely to be a case study supporting http://xkcd.com/882/.

But the research does have a questionable assumption. On p 5 it says, "Such a result would not likely be due to differences in the methods used to generate g in the various datasets under comparison, since g is consistently measured by a wide variety of tests."

This is true. However it is also true that, as http://vserver1.cscs.lsa.umich.edu/~crshalizi/weblog/523.htm... points out, that all evidence we have is consistent with g being a correlation between different ways of weighting a large number of independent factors. If so, then it is easy to have two test measuring g that are fairly well correlated with each other, one of which picks up a particular factor and the other does not. If a gene affects that factor, then the test you use to measure to find these correlations will matter.

View on HN

> This whole field seems likely to be a case study supporting http://xkcd.com/882/.

The scientifically interesting thing is that it's not at all like that XKCD cartoon. Multiple hypothesis testing is built into the design of the studies, and has been from the very start. When it comes to proper interpretation of stats in science, geneticists are at the top of the hierarchy.

Your second point is far closer to what I think is going on, but I still think that single-gene studies are the wrong way to go, as you might guess from my username. They're testing for the wrong thing, single locus or single gene causes, of a very complex phenotype with the involvement of many many different loci and genes. So if there are 500-600 genes that are important, maybe one passes the test in your particular subsample, but in a different subsample the rest of the 500-600 genes mediate the effect. I.e. there's not enough complexity in the model to account for the signal. What's worked very well for simple diseases and traits may not work well for complex traits with the influence of many many different genes.

Multiple hypothesis testing is built into the design of the studies, and has been from the very start. When it comes to proper interpretation of stats in science, geneticists are at the top of the hierarchy.

It is built into the design of the studies. But studies that do not find results do not get published. Thus there is a lot of room for exceptional results to sneak through despite the attempts to get the stats right.

GWAS studies are extremely expensive studies, often in the hundreds of thousands of dollars in SNP chips alone, and will get published somewhere, somehow. If they don't find results they get published in lesser journals, because there's always something to be said even if there are not significant associations.

It's the same with academic clinical trials, even negative results get published, because for those academics they still need something to show, and a publication of a negative result is better than nothing.

Non-publication bias is more common in less expensive, more exploratory settings, where there are lots of little experiments going on all the time. This is quite different from GWAS.

Also, for anyone reading Shalizi's article, please note that he uses the term "statistical myth" in a non-standard way. By his definition, any aggregate quantity derived from a microstructure model (e.g. temperature, pressure, climate, electric current) is a "statistical myth".

This is not to dispute his calculations or model - they seem solid. I'm just pointing out that his definition is probably not what most people think when they hear the term.

I don't know enough about STATs to get the XKCD comic. Anyone care to explain?

Something about how P < 0.05 still doesn't mean there is a link between green Jelly Beans and acne?

P < 0.05 means that the probability of seeing the observed results if there is no difference is less than 1 in 20. So, if you run the experiment 20 times, you'd expect to see a link once, even if there is really no link.

Reply on news.ycombinator.com