Statistics as algorithmic summarization
benjamin-recht.github.io
benjamin-recht.github.io
What are your thoughts on that mask example? To me, this seems like a reasonable critique, but (as I just posted in this thread) I don't have a deep understanding of statistics, so I am a little uncertain if my interpretation of the blog post is correct.
"So what is the remedy here? The thing is, we already know the answer: if we randomized the assignment ... it is critical to decouple the randomness used to probe a system from the randomness inherent in its system itself. Statistical models are not necessary for statistical inference, but randomness itself is amazingly… let’s say… useful for understanding natural phenomena."
This makes sense to me if there's always a population that you can random sample. And yes, you'd have to sample a lot for the sample and population summaries to converge, but this seems fine in certain contexts (i.e. when you have access to computer simulations). Your counterpoint seems correct when you aren't able to randomly sample the population in this manner.
Would you agree with the author based on the specific framing I'm making (randomly sampling population beats building a probabalistic model if you are able to do the former). Again, I'm a stats novice so apologies if I'm making an obvious point.
This is a silly mistake to make in an article on statistics. Any statistician worth their salt can see the problem with an operational definition of height that starts with "bottom of their feet".
I like this article, but then it goes on to miss another very critical part of the explanation:
> What if we selected a subset at random, and used this subset to estimate the mean? That is, we could collect a random sample of individuals from the population and measure the average height of all of the individuals in the sample.
Why would we select a subset at random? Why shouldn't we try to select a representative subset? We know from experience that biological sex, country of birth, and affluence of family are three strong predictors of height. Why shouldn't we deliberately select a subset that balances out these variables in proportion to their population?
There is a good reason to do a random sample rather than try to construct a representative sample, but the article just brushes past it!
I don't get it, what's the problem with that?
Either one of those methods are fine, but leaving it unspecified is a threat to external validity.
Isn't this similar to the point this article is trying to make? Looking at statistics as a collection of algorithmic recipes shifts attention to how those procedures are designed, and when do they yield us useful summaries. When you want a "representative mean" rather than the population average, you'd just tweak your algorithm.
The problem with representative samples is that you can't know whether you've actually constructed one, and even if you have, you'll find it hard to compute the errors of your estimations.
My beef is not with thinking of traditional frequentist-objectivist statistics as a set of algorithms solving specific problems (because that's what it is), my beef is with not starting out explaining exactly when the algorithms apply and why.
When you take a black boxy algorithmic approach to something, you have to be particularly clear about things like that.
> There is a good reason to do a random sample rather than try to construct a representative sample, but the article just brushes past it!
> This is a silly mistake to make in an article on statistics. Any statistician worth their salt can see the problem with an operational definition of height that starts with "bottom of their feet".
You missed the forest for the trees.
> Why would we select a subset at random? Why shouldn't we try to select a representative subset? We know from experience that biological sex, country of birth, and affluence of family are three strong predictors of height. Why shouldn't we deliberately select a subset that balances out these variables in proportion to their population?
Random subsets are core to statistics, and they are independent of representative samples. You choose random subsets so that you can compute statistics with well defined confidence intervals (as in the less randomness there is in your sample the poorer/hard to quantify your confidence would be).
You use representative samples when you want to compare groups. Representative samples should still be random per group.
I guess that an even better predictor is the parents' height.
I'm not sure how the numbers play out but would expect that to be a big confounding factor given changes in global poverty.
Here's my best interpretation thus far:
This particular post is a continuation of his two earlier posts[1], [2] which seems to critique the way statistical significance is simplistically calculated from a gaussian distribution when effect size is small. This leads to an argument that the statistics community is too dependant on generative models for their interpretation, given that these models are typically to simplistic outside of hard sciences:
> "But in biology, medicine, social science, and economics, our models are much less accurate and less grounded in natural laws. Most of the time, models are selected because they are convenient, not because they are plausible, well motivated from phenomenological principles, or even empirically validated. Freedman built a cottage industry around pointing out how poorly motivated many of the common statistical models are."
This part I am a little unclear on, but it seems like this leads him to suggest focusing on the random sampling as a way to get counts that you can then plug into various statistical formulas, without the need to assume a probabalistic model:
> "So what is the remedy here? The thing is, we already know the answer: if we randomized the assignment, we can estimate log odds by counting the number of positive outcomes under treatment and control, and then just plugging these values into the odds ratio. If you do this, you find an estimate whose median is precisely equal to the true log odds. No covariate adjustment is required."
So I think, what the author means by "algorithmic summation" is that we focus on random experiment design, and discard model assumptions. Is that right?
If so I think this makes sense. I believe this is something Allen Downey has talked about before, specifically saying statistical experiments can now take advantage of cheap computational simulation to hit the large numbers needed for the sample to approximate the population, without a need for the typical model approximations developed in a pre-computational era. Downey's post here: http://allendowney.blogspot.com/2016/06/there-is-still-only-...