“Mindless Eating,” or how to send an entire life of research into question

> a data set is not completely worthless if it showed a null result.

Why is a null result worthless? I don't understand. It goes against my common sense that paying more for food doesn't make you eat more. The fact that this is not the case is valuable knowledge to me. I understand that it's not as glamorous as "paying more makes you eat less!", but it's still valuable knowledge that should be published.

Even the post-hoc analysis isn't invalid as a way of discovering potential avenues of future research. But you really have to run new experiments designed to test your new hypotheses.

> a data set is not completely worthless if it showed a null result

As the other commenters have pointed out, the fact that it showed a null result is not really relevant to future utility.

A better way of phrasing it might be "A data set is not completely worthless after it has been used to the test the hypotheses for which it was gathered."

As to how it is useful, I think a sound summary would be "a data set is _only_ useful for validating (or invalidating) those hypotheses for which it is gathered, but it may be useful for deriving new hypotheses for which new data can be gathered to validate."

Or more negatively as "a data set is not useful for validating or invalidating hypotheses invented after the data set is seen, but it may be useful for deriving those hypotheses and leading to further research."

To some very limited extent, if a mechanism of action is established (which is to say there is an external reason to infer a correlation) then cross-validation with an existing data set may be able to derive the parameters of that correlation.

It seems like a nothing to me. He didn't say, "publish any thing with p>0.05," but rather, "see if there's something interesting."

> Meanwhile the knowledge of what works is blatantly obvious: the energy expenditure must be higher than the intake.

This is like saying about the poor, "What works is blatantly obvious: Poor people just need to increase income or reduce expenditures. Boom! Poverty solved."

The secret is that this isn't the problem. The actual problem that fat people must solve is not what to do, but how to do it, both in the short- and long-term.

> Having a healthy diet and exercise regimen along that is highly desirable but not strictly necessary...

1,000 calories of cupcake are 1,000 calories of chicken breast are, in theory, the same amount of energy. In practice, diet and exercise considerations are integral aspects of the actual problem that fat people must solve.

Example: For some, exercise is actually counter-productive and causes more hunger. For others, exercise is a crucial part of success.

Example: For some, calorie restriction can be done with no particular attention paid to the balance of carbs, proteins, and fats. For others, a low-carb diet means lower hunger/higher satiation for longer and is a must for success.

> the energy expenditure must be higher than the intake

I think you might be overestimating the predictive power of this statement. The reason that these fad diets and questionable research end up being so popular is that people have a huge amount of anecdotal evidence that this does not apply as a necessary condition for weight loss.

I think it would be hard to argue that it isn't sufficient; assuming that "energy expenditure" and "intake" are well-defined (note that even the model of "calories" in food is not a chemical notion, as many think, it's a heuristic based on a known incorrect model of metabolism), but people see evidence all around them (once again, anecdotal, and in some cases based on incomplete evidence) that people who neither restrict input nor expand expenditure can maintain a healthy weight.

People want there to be a magic bullet that will let their metabolism shift into a state where the body would simply stop storing excess calories as fat. I'm fairly convinced that we haven't yet found it, but I think it is a laudable goal. And, because losing weight has positive health and social implications, this is a place where people are desperate enough for a solution that won't involve unpleasantness that they're willing to believe unreasonable things.

For what it's worth, this article is not about "$name diets." Rather, it is about a researcher's use of questionable methods. The researcher's field happens to be psychological factors affecting eating behavior, but his questionable methods can be/are used in other disciplines (as pointed out in the article). And based on the summaries in the article, the researcher seems to be in complete agreement with your rant: his studies are not about what to eat, but how to more easily achieve the intake vs. expenditure balance (for example, by using smaller plates to make it psychologically easier to limit portion size, or giving low-calorie foods a name that makes them more psychologically attractive). So, it's hard to see what in the article this rant is responding to.

> Moreover, you won't find any reputable study on the web where the average person lost 10%+ of their body weight and kept it off for five years. Not even one.

Of course there is: http://www.nwcr.ws/Research/published%20research.htm.

Here's a good question: other than the fact that these people lost weight, what is identifiably unusual about them?

To contradict this a little bit (and this is not 100% relevant because it involves exactly one participant, so who knows whether it would ever work for anyone else) this remains one of the most interesting weight loss related things I've read on the internet: http://edwardjedmonds.com/wp-content/uploads/2013/12/Stewart...

The study participant went from 456 to 180lbs through the course of a year, and when they checked on him five years later, he weighed 196lbs. Obviously this is an extreme case, but it is one tiny and yet super interesting datapoint on the subject.

This is very depressing :(

As someone trying to lose weight myself, calorie tracking has so far yielded about 5% weight loss over a period of several months but I'm realizing that if I stop counting calories there's a strong possibility of gaining that back. This just lends some credence to that idea.

Weight loss is difficult because dieting isn't a train you get off once you're at your target weight. It's basically a way of life to maintain your target weight. I don't know too many people that are capable of rewiring themselves to that kind of degree for the long-term

Honestly, the human body just isn't built to lose weight. We're built for starvation and packing on as much as we can, while we can. Humanity is just a victim of its own success with the sheer plentiful amount of food that's available now, at least in the first world.

Working as intended, WONTFIX, etc. Obesity at this stage of our civilization is not abnormal. It sucks, but it isn't abnormal.

It's interesting that the comments on any article having anything to do with food automatically become a "weight thread." The topic of this article is poor scientific methodology and the failure of the current peer review system to address it, not diet or weight loss. And (based on the studies summarized in the article) the researcher discussed in the article works on psychological and economic factors in eating behavior, not defining what people should eat or promoting diets.

This was my thought exactly.

"Wansink added another note to his blog post, acknowledging the problems and announcing that a statistician would be redoing the analyses."

With all of the issues lately around replication of studies it seems like having a statistician review the math should be step one of any peer-review process.

Interestingly, a few years ago Nature planned to include statisticians...

> as consultants on the rare papers in which there are serious issues about the statistical techniques used, at the editor's discretion and based on the referees' suggestions.

I think the crux is that editors already have enough trouble finding reviewers, and see getting a statistician as often unreasonable. It's too bad, because the nature of statistical issues is that people tend to be unaware of them until a statistician (used broadly) points them out.

> In fact, without open data ... it would be impossible for peer reviewers to validate any numbers

Even this is not sufficient -- the fact is that this is a methodological statistical error, not a mathematical statistical error.

The mistake is thinking that a dataset alone can yield data supporting a hypothesis derived from that dataset, rather than deriving a testable hypothesis from an existing dataset, and then gathering new data to validate.

I would say the only way that you could publish results derived from an existing data source is if you also published all the null results you got along the way when examining the data set; but this is not really a feasible thing to print in a publication; either the list would be very long and thus it would be clear that some sort of p-hacking was involved, or the list would be short and accidental or intentional omission would be suspected.

This dramatically reduces the amount of useful results that can be squeezed from a dataset, which is unfortunate, as many of them are hard to gather in the first place. It might be necessary to protect these datasets better -- to restrict access conditional on specifying the hypothesis being tested with the requirement that all results, including null results, be recorded, even if only in summary form if the results are not interesting enough to be accepted for publication or to be worth the effort of composing into a quality research paper.

This doesn't even begin to get into the effects different macronutrients have on you. It's a complicated subject that has reached religion-like levels of zealotry.

Yes, but if we started acknowledging that everyone's different and functions on their own level, people will stop giving statisticians money! Who wants to live in that world?

[/sarcasm]

Wansink strikes me as an extremely bright, creative researcher. His problem seems to be that he didn't collaborate with people who were strong methodologists.

If anything, he reminds me of a lot of brilliant researchers, who are too confident in their methodological prowess. (And this is the outcome that should have been avoided through collaboration).

Well that's a lot of data you have there. I did some GLM analysis and clearly weight is correlated with gender. Let me go tell some journalists.

> Isn't that fraught with survivorship bias?

It is, but it's not a completely invalid starting point. Even better is to study people who were once fat, but lost weight and keep it off. This is what the National Weight Control Registry does. It's similar to studies that have tried to pin down why marriages fail. Most studies focus on what went wrong in failed marriages, but some of the most helpful suggestions have come from what people who stay married do.

> My son and I are normal weight. My wife and daughter are overweight.

Do your wife and daughter eat the same portions as you and your son? It's a common complaint of women, who are on average shorter than men and therefore need fewer calories, that they can't eat as much as their significant others.

I was just about to ask, isn't that how data science works?