Hacker News Reader

Hacker News

Comment by yxitcti | Hacker News Reader

Full thread

yxitcti

I long for the day when these correlation-digging studies with absurd claims will stop.

https://xkcd.com/882/

View on HN

You should take a look at the journal article. They comment on how they corrected for multiple testing

"In addition to MANOVA, the Bonferroni procedure was used to protect against multiple comparisons."

The Bonferroni correction is one of the most conservative ways to correct for multiple testing.

https://en.wikipedia.org/wiki/Bonferroni_correction

The study is still 'correlation-digging'. Bonferroni won't fix the problem that these kinds of studies will throw up statistically significant associations that provide no insight into anything meaningful, due to confounders, biases, poor design etc. There is no equivalent xkcd however.

I don't think that anyone would take the results of this article as a statement of absolute fact. In the discussion they mentioned some more (better controlled) studies investigating the role of cocoa flavanols in "cognitive tasks". Doing any longitudinal study with humans suffer from confounders, biases, and poor design because we can't do controlled, long-term research on humans. You just have to take these limitations into account when you are interpreting the results and use any findings as areas for possible further investigation.

Yes, that's fine, I was pointing out that correcting for multiple hypothesis testing does not deal with the issue cited (albeit rather abruptly) by yxitcti.

There's more to it than just the things you mentioned, and I'd argue that it's an even bigger problem. The fact studies are adaptively chosen and performed based on the results of past studies on the same (or related) data leads to increased instances of false discovery. This is something that is directly addressed in some very recent recent work (not uncoincidentally making reference to this exact XKCD comic):

http://googleresearch.blogspot.com/2015/08/the-reusable-hold...

Freedman's paradox refers to adaptive data analysis within one dataset. The reusable holdout is designed to permit adaptive analysis on the same dataset without inflating the false discovery rate. The problem doesn't apply if you are collecting new data and testing the hypothesis, that's the ideal scenario.

Yeah, Freedman's paradox is definitely part of what I'm alluding to. The other aspect is that for widely studied phenomena, some groups are likely to come to wrong conclusions just due to statistical error. E.g., if 100 groups independently investigating whether "Eating chocolate is associated with improved brain function" (and we'll assume it's false, just to make my point), a few are likely going to come to the conclusion that it's true. Even worse, some studies with "expected" results may not be published (either by choice of the investigator or by the reviewers); instead, the "interesting" or "exciting" results will get published.

very true. more specifically, you generally find what you are looking for. much like trying to google if you have some sort of rare disease.

people should pay attention to how they feel 30 minutes after eating a particular food more than any research study that has ever been published.

Can we just have this one? It's chocolate, man.

Sure, eat your heart out. Just try not to eat your heart out.

Correlation-digging studies absurd or otherwise have led to an awful lot of peer-reviewed literature detailing a raft of <<molecular mechanisms>> by which flavonoids may be implicated in improved brain function.

To pick one at random: "Flavonoids exert a multiplicity of neuroprotective actions within the brain, including a potential to protect neurons against injury induced by neurotoxins, an ability to suppress neuroinflammation, and the potential to promote memory, learning and cognitive function. These effects appear to be underpinned by two common processes . . ."

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2593006/

Most of us enjoy eating the stuff so it's no surprise that chocolate as a flavonoid-containing substance gets special attention.

Can someone please explain to me the difference between p-value and the confidence interval?

Reply on news.ycombinator.com