Comment by captainmuon | Hacker News Reader

Comment by captainmuon | Hacker News Reader

Parent Full thread

I find this whole backlash against p-values pretty confusing. That is probably because I come from particle physics, where we also use a lot of statistics, but in subtly different ways.

Hypothesis testing is not too hard [1]. You pick a cutoff, say p < 0.003 ("3 sigma"), and then if your p-value is below that, you call it evidence for your signal - otherwise you just don't have evidence. By doing so, the probability is 0.3% to have data this or more signal like, assuming there is no signal. With other words, if you follow this prescription and are looking for something that isn't there, in 0.3% of cases you will (wrongly) claim evidence (error of the first kind).

Since we are a cautious bunch, we actually put the threshold for discovery at 5 sigma - p<0.0000003 - which sometimes gets us ridiculed by statisticians. This hyper-strict standard shouldn't be necessary, but in part it's a hedge against the case where you get your systematic errors wrong (you believe your prediction for the null hypothesis is more accurate than it is - so if you see a slight fluctuation, it seems to be many (wrong) standard deviations away).

One other thing that we have to take into account - and many people forget this - is the look-elsewhere effect. If you perform one search, looking for e.g. a Higgs Boson with a mass of 126 GeV, you expect N events in your experiment if it is not there, and N+X if it is there. You know how N is distributed, and the interpretation is straightforward. However if you perform a scan, looking at 120, 121, 122, 123... GeV, then you have to adjust your p-value, since you are basically performing a bunch of different experiments, and by chance alone some of them are bound to turn up "significant".

The same thing applies when hundreds or thousands of Master and PhD students and postdocs do their analyses - even if no one makes a mistake, some of them will "find" a 3 sigma or larger effect that isn't there, just due to the sheer number of independent statistical tests performed. I've "found" new particles myself this way, but when you keep calm, put it into context by looking at other analyses, and try to add more data, you'll often find that your result melts away.

------------

[1] explaining it is hard, and I will undoubtedly have messed up, especially since I'm tired.

The interesting problem is that if you set a strict significance threshold but have a sample size that is too small, you will still sometimes correctly get significant results, but the effect sizes will all be exaggerated.

If the sample size is too low, the only effect size big enough to be significant may be one that is much larger than the truth. So you only claim significance when you get lucky and exaggerate.

This is actually an enormous problem, and it probably affects physics too. Many medical and biological papers tout huge effects which turn out to be completely wrong -- not because their result was a false positive but because it's an exaggerated true positive. Sample sizes tend to be hilariously inadequate in soft sciences where each data point costs thousands of dollars.

I call this "truth inflation," although I don't know if it's been discussed enough to have a common name. It's heavily discussed in my book, if you're interested in seeing why the backlash against p values is so widespread: http://www.statisticsdonewrong.com/regression.html#truth-inf...

> I find this whole backlash against p-values pretty confusing.

Plenty of research (not in particle physics) uses the cut-off of 0.05 instead, misunderstands the p-value as "probability that the result is not real", and also ignores the fact that a p-value of 0.05 can too often be reached by making a variety of choices in experimental setup and data analysis. When the headline calls it a "Tool to Sift Research Fudge from Fact", NHST wasn't doing this job well at all, and wasn't really designed for this job in the first place. There are far, far too many people who seem to think that p<0.05 means something is a "research fact", which is a failure of statistics education as well.

I studied physics too, and am not a statistician.

My way of interpreting an "absurd" threshold such as 5-sigma, is that you can stop using hypothesis testing, and start using other tools for interpreting data.

Another interpretation is that we set these huge limits because, deep in our hearts, we aren't really comfortable with hypothesis testing. There's Rutherford's famous quip: If your experiment needs statistics, then you should have done a better experiment.

I think statistics are actually used pretty sparingly and with caution in physics. There are lots of experiments where the presence or absence of a phenomenon is not the primary question. I didn't report a p-value at all in my thesis project.

> which sometimes gets us ridiculed by statisticians

You get laughed at because you have this huge luxury at dissecting huge amounts of data, running the experiment billions of times.

Do you understand that in most disciplines, you don't have that luxury?

To begin with, you'll have to explain to the physicists that there are other disciplines besides physics.

Sample size. Also particles are "the same" and much simpler.

Put a particle in a magnetic field and it will behave the same every time (and you can usually do this 1e3, 1e4, 1e5 times as needed). Give 100 people the same drug (ok, 50 of them get the drug, 50 get the placebo) and see each one have a different reaction.

Actually the whole point of particle physics is that the same thing _doesn't_ happen every time. Quantum field theory allows us to predict probabilities e.g. of a particle decaying to another particle, for a given model. As for doing something 1e4, 1e5 times, at the LHC we collided bunches of 10e10 particles continuously every 50ns for months, and still only produced a few hundred Higgs particles. So these studies of ultra rare processes tend to also have limited statistical power.

It's still considerably less than the variability in sciences based in biology, though.

Yes, in quantum physics you have a set of possibilities and their probabilities, and with more experiments you'll produce more Higgs particles, for example

> we actually put the threshold for discovery at 5 sigma - p<0.0000003 - which sometimes gets us ridiculed by statisticians

This says more about the sillyness of statisticians than that of particle physicists.

Your explanation worked really well for me. I now understand what it is bayesians don't like about p value reliance.

Reply on news.ycombinator.com