Hacker News Reader

Comment by Fede_V | Hacker News Reader

This is a difficult topic to study properly, because:

- Coffee consumption is not uniform throughout the population. For example, I'm willing to wager that people that eat a lot of truffles and Beluga caviar are among the healthiest and longest living people in the world. Anyone care to wager why?

- Long term controlled studies on human subjects are tricky. Can you imagine taking 100 people at random (not volunteers!) and forcing half to drink coffee, and half not to drink coffee for 30 years?

- Plausible mechanisms are always incredibly easy to postulate since we still have lots of things we do not understand about the human metabolism.

Unless the effect is very, very strong, observational studies are going to be rife with false positives and spurious correlations due to Simpson's paradox. Ignore this awful article, and almost all dietary advice in this vein.

View on HN

These are fine points, but if we factor in the probability that researchers carrying out these studies took them into account, we don't need to ignore "almost all dietary advice in this vein."

I would agree with this if researchers were not under insane pressure to produce findings (as opposed to understanding).

If a researcher designs a great experiment, but after collecting and analyzing all the data comes to the conclusion that there is no link between eating chocolate and health, then she will a horrible time publishing it in a respected science journal, even if that's what the data says.

I wonder if this is a dogmatic pressure or a financial pressure? Second could probably be solved by increasing public funding but not sure about how one would approach the first.

There are a billion things about which you could publish a negative result, and it's hard to judge whether a negative result is particularly interesting. On the other hand, positive results are fewer and thus a priori more interesting than negative results. They also stimulate follow-up research much more than a negative result.

Since journals want to publish interesting articles more than boring ones, and articles that are likely to be cited more often than articles without follow-up results, there is a bias against accepting negative results. As scientists are judged by the number of things they get published, and the number of citations their papers get, they have every incentive to only publish positive results.

I strongly disagree. The vast majority of published articles I stumble across conclude with "we didn't find anything". They are published in journals, but almost never make it into The Times. I'm definitely not saying research pressure doesn't exist or doesn't have an impact on research quality, but I think selection bias is hard at work here.

"We didn't find anything" is fine for a random journal (there is a saying in academia that everything gets published eventually), but that's not going to get you into Cell, Nature, or Science, or a similarly top-tier venue. That's where the stress lies, and why negative results are pointedly lacking in the literature.

Coffee consumption on health is probably one of the easiest things to study, because it's so easy to collect big data on it and perform robust statistics. My point is that coffee consumption is so common and widespread among different populations and genetic backgrounds that we can get very big sample sizes, and sure there can be confounding factors, but with statistics we can adjust and control for that. Then it's a whole other thing to explain the biochemistry behind something like coffee consumption and health, that will require other data and experiments. There is a clear publication bias in science toward positive results, however, if a large, well-designed study running over many years come up with a negative result - this is actually not hard to publish at all.

> there can be confounding factors, but with statistics we can adjust and control for that

The extent to which you can adjust for a confounder depends not just on your sample size but also on on whether you were able to measure that confounder without bias. Big samples don't reduce bias.

Numerous examples exist of people who are dead certain that they've accounted for every possible confounder and that they've measured those confounders in exactly the right way... except they didn't.

My favorite example of this is the benefits of modest alcohol consumption. The group of people who don't drink include those who abstain from alcohol for medical reasons (former alcoholics or medication that doesn't mix well with alcohol) and so of course non-drinkers are going to be less healthy on average than someone who likes a drink every once in a while. It sounds obvious when I state it like this and you would figure every questionnaire and analysis would take it into account, but study after study after study investigating the effect of alcohol on heart health over a timespan of half a century failed to take this possibility into account.

Cf. http://www.annalsofepidemiology.org/article/S1047-2797(07)00...

(There are similar concerns about e.g. the risks associated with red meat and the benefits associated with breastfeeding, two other scientific findings that have become conventional wisdom but that in fact we are still uncertain about.)

Feynman has a diatribe about this, regarding psychological studies and rats in mazes.

And because coffee consumption tends to correlate with processed sugar consumption (which brings in its own list of adverse effects). And with various bad habits (overworking, lack of sleep, poor nutrition) that people tend to mask with their caffeine intake.

I'll definitely agree with you on the sugar consumption. Any benefits that may or may not exist are sure to be dampened/counteracted by excess sugar intake. It's kind of like eating salad to be healthy, but then drenching it in high calorie dressing.

Reply on news.ycombinator.com