Hacker News Reader

Comment by jerf | Hacker News Reader

I've been grousing about the state of science lately in some other comments; this is basically what I was talking about. It's still the best method to the truth we've got, but it needs some upgrades.

One other big error I'm reasonably confident about (though I'd welcome corrections from those who know more stats) is that the p-value, in addition to its other faults and misinterpretations, is usually used in a manner that assumes a Gaussian distribution. While the Central Limit Theorum does mean we tend to see that more often than some other distributions, it is not true that it is safe to simply assume your data is Gaussian. You really need to demonstrate that it is, first, then you can start using Gaussian-based tools on the data.

View on HN

I am wary of trying to contribute to this (a little knowledge is a dangerous thing) but hopefully someone can correct my mistakes.(And what better way for me to learn...)

A typical controlled science experiment is designed to take measurements of multiple groups where one variable is different between the groups and others are controlled. We wish to see if the variable of interest has an effect.

Therefore the commonest statistical test is to determine if the mean value of the groups are different (t-test for two groups, anova for multiple groups) e.g. does mean blood pressure increase when on a high salt diet.

If I understand the CLT (big if) then the distribution of the _mean_ of a sample is, by the CLT, going to be Gaussian, regardless of the distribution from which the actual measurements are drawn. i.e. for comparing group means it it doesn't matter if my data is sampled from Guassians or not.

Of course that leads to the question of if a significant difference in group means is really relevant in a given context.

"If I understand the CLT (big if) then the distribution of the _mean_ of a sample is, by the CLT, going to be Gaussian"

Yes, for an infinite number of samples. The rate at which is converges to a Gaussian, though, is strongly dependent on the distribution from which the measurements are drawn.

Actually, not even this is not true!

The Central Limit Theorem applies only to samples which have finite mean and variance (see: http://en.wikipedia.org/wiki/Central_limit_theorem).

Take a distribution which has infinite variance or mean and you can wind-up instead with one of the fractal distributions which Mandlebrot studied.

see http://en.wikipedia.org/wiki/Stable_distribution

Distributions with infinite variance or mean are more common than one might imagine. Some might argue the stock market would qualify.

Good point. Distributions with infinite variance are less common in physics, so that's normally not something we have to worry about.

You need an infinity of samples to be sure.

But moreover, in most situations, the other variables cannot be absolutely controlled but rather are only roughly controlled. How roughly depends on the field and is often the rub.

Absolutely true, but this is more a criticism of experiment design than statistical methodology.

Also it is bias in uncontrolled variables that is most concerning. (More than uncontrolled noise although that will reduce the power to detect real differences).

This is also only true if the distribution of each sample is the same. If you don't understand the system you're sampling from, or where the variability comes from (maybe its systematic), then all bets are off.

Not even necessarily true then - you also need each sample to have finite mean and variance...

Reply on news.ycombinator.com