Statistically Controlling for Confounding Constructs Is Harder Than You Think

This is really interesting. Trying to summarize the idea of the paper using their example: If people swim more on days when ice cream sales are high, one might conclude that ice cream causes swimming. But if you have information about the temperature, you can fit a statistical model that predicts swimming. The model will learn that temperature predicts swimming perfectly, and the feature of ice cream isn't used at all. So you could say after controlling for temperature, ice cream no longer correlates with swimming.

But what if your temperature estimate is noisy or imprecise? For example, if you only have information if the temperature was hotter than 75 degrees, but not exactly what it was. Then this trick no longer works. The temperature variable predicts swimming a lot, but not completely. So ice cream sales are still useful, and so still correlate with swimming after controlling for temperature.

I wonder if the way to solve this is using bayes nets. You could test a model where the variables "swimming", "ice cream sales", and "observed temperature", are all caused by some unobserved third variable. This model should fit the data better than alternative models, like ones where "ice cream sales" cause swimming and temperature. But simple linear models can't represent dependencies like this.

I always wondered what exactly people mean when they say “controlling for ...” as if somehow they can tell apart cause A and cause B for the effect C.

It turns out if you can measure A and B without noise, the whole "controlling for..." business is legit, but if you only have noisy estimates A' and B' of the actual A and B, then the "controlling for..." technique used by many researchers does not work very well at all.

interesting quotes:

> The present results demonstrate that, for a very common class of incremental validity arguments, such a strategy runs a high risk of failure. The scope of the problem is considerable: literally hundreds of thousands of studies spanning numerous fields of science have historically relied on measurement-level incremental validity arguments to support strong conclusions about the relationships between theoretical constructs.

> We conclude that many previous findings in the social sciences are at high risk of having concluded that two constructs are distinct when they may not in fact be so.

Note how polite the authors are. They don't say it straight up that many (most?) multiple regression results are bullshit, they just say "at high risk of being wrong."