Hacker News Reader

Comment by knzhou | Hacker News Reader

Duplicate of this post. [0]

I have the same comment as over there:

> It's important to keep in mind that there are always plenty of outstanding experimental anomalies in physics. At the moment, this is one of ~40 roughly equally credible hints towards new physics, and it's more likely than not that all of those hints will fade away over time. That isn't anybody's fault either: it has always been like this, and it happens because experiments are difficult and subtle.

0: https://news.ycombinator.com/item?id=21616084

View on HN

Why do they fade away? Because the experiments have been misinterpreted?

Because they're statistical artifacts, basically. P=0.05 means you're got a 1/20 chance of it happening on any given random test set where the bill hypothesis is true, after all. Physics uses far higher confidence levels, but also runs (effectively) massive numbers of tests. That's part of the danger of picking theories based on the data - each degree of freedom in interpretation multiplies your chances of having spurious results crop up.

Given the high statistical confidence they have, 7.2 sigma or 1 in 2 trillion, it's more likely to be some sort of systemic issue with the experimental setup.

This is well known and corrected for in properly conducted experiments.

https://en.m.wikipedia.org/wiki/Multiple_comparisons_problem

I almost agree. This is the more probable explanation of the other ~39 anomalies that are at 2 or 3 sigma. As a rule of thumb, don't get too attached to a 2 or 3 sisma because there is a high chance that it is only a statistical fluke and the apparent result will disappear with more data.

But this cases has too many sigmas, 7!! And the group has a history of dubious discoveries, like the 5 sigma result in 2015 that was never reproduced. So I'm more inclined for a bad measurement method until an independent group can replicate it.

> like the 5 sigma result in 2015

Isn't that just an earlier version of the same experiment? Doesn't seem too surprising that a broad distribution has a mean shift with a different experimental setup. I'd count this as a repeat of the same experiment. That of course doesn't rule out some flaw in their experimental setup that happens in roughly the same way in each repeat.

The mass of the result of 2015 is close to the current claim, so I have to correct may comment and say that it has not disappeared (yet).

But they have a 13.45+-.30 MeV result in 2012 with a significance of 3 sigmas [1] and a 12.0+-2.5 MeV result in 2008 (I can't find the significance, it looks like a 2 or 3 sigma) [2].

If it possible to get a false result by chance like 1/20 or 1/100 of the times, but they already got 2 of them. So I'm very skeptical of the new third claim.

(Also the story about the protophobic particle is strange. There have been more strange things, but this is strange anyway. And it's also not clear why the protophobicity hides the particle in all the previous electron-positron collision experiments in many other laboratories.)

[1] https://www.lnf.infn.it/sis/frascatiseries/Volume56/Krasznah...

[2] http://www.actaphys.uj.edu.pl/fulltext?series=Reg&vol=39&pag...

Just to play advocate more, if you look at the broad excess in the angular plot in all those papers it shouldn't be surprising that this results in a broad and changing best estimates for mass. So without crunching the numbers, again its possible all these are measuring the same thing with a fluctuating best guess for mass. I think we shouldn't dismiss the latest result due to the previous results. I think we should wonder what could be the systematic issue that could cause this effect without it being new physics. I think the excess in all the experiments will likely have the same root, be it new physics or something else that is not yet understood. So don't count those previous papers against them, those are all measuring the same thing and the convention of physics to match to some theory is causing the precise conclusion to jump around. If you look at the actual raw data the effect is not changing too much.

The relevant graph are in the Figure 6 of the first paper and Figure 2b of the second. The idea is to match a simulation of the results you would expect if the new particle has different masses and compare it with the experimental data and see how good is the fit. The horizontal axis is the mass of the hypothetical particle in the simulation, and the vertical value is how good is the fit (lower is better). In both the 17MeV region is out of the interesting region. So if there is an effect, they did a very bad analysis before.

I think it's better to wait until another group can independently reproduce the results. It doesn't look too complicated, for someone that has a similar lab.

(It's not like the results of the LHC, where the reproduction steps start with: "First dig a 17mi long circle and then call for further instructions." :) Note that to try to avoid false positives, they have two almost independent groups in oposite points of the circle.)

Yes. They were never really there in the first place.

I'm a physicist and I've experienced this on the small scale of my own humble work developing measurement equipment. When you observe a surprising effect, you try to make it go away. First, you check to see if it's reproducible under a variety of conditions. Then you search for sources of bias and error that you might have overlooked. You remove things that you think should be extraneous to the purported effect.

At some point, you're satisfied enough, and you publish. Then the rest of the community goes to work trying to do the same thing, with more care, possibly more money, and more minds working on it.

Thank you. This an excellent succinct explanation of how unknowns get chipped away over time.

Decades ago, when doing physics and programming together, I discovered the parallel between my experiment revealing an anomaly unexplainable by the current scientific theories and my code revealing a bug in the current version of the compiler. The most likely eventual outcome was not fame and glory but more likely, "Oh! Oops. Never mind...."

We should never underestimate the likelihood of a surprising observation being explained by a silly mistake or some mundane chance event.

Like you, I've had this experience countless times when debugging a piece of code, usually my own :)

It's never a compiler bug, but when it implies new physics, it is more probably a compiler bug

The famous superliminal neutrino case is a great example of this process at work.

Also the Pioneer anomaly. From 1998 to 2012 it hinted to new physics (alternate gravity laws) when the Pioneer 10 and 11 probes flied outside the solar system. Then in 2012 it was solved by accounting for the multiple scattering of heat radiation, as the probes are not convex bodies.

https://en.wikipedia.org/wiki/Pioneer_anomaly

The Pioneer thing, I should collect on some bets. I was on the side of "not a new force, it's some kind of radiation pressure that we're overlooking by treating this like a sphere." Not long after I mentioned that, I attended a lecture by a guy who spent a lot of time working over Casimir effects on strange topologies yielding equally unlikely results. He said something to the effect of "never underestimate the numerous small anomalies created by minuscule forces on complicated surfaces."

I've been watching people look for a fifth force (and sixth) force in different flavors for decades. Maybe there's more out there but we have to work very hard to eliminate all of the numerous potential sources of error. If I had to lay a guess on it, it might take centuries of work find an additional force were it present.

I really think they should have known better, and done more diligence before announcing the anomaly. Superluminal anything implies clock skew issues, and iirc they didn't methodically eliminate those before going public. I don't think the director would have needed to resign if they hadn't taken the bait so eagerly.

Except that this is the replication.

Well, to paraphrase what Bellarmine wrote to Galileo, if it turns out to be true, then we've got some cogitatin' to do.

One route is to try and beef up the results, for instance by looking for similar results under different conditions. Then, if someone finds a pattern that ties these results together, there's the chance that they could write a theory. The big win is if theory and experiment can play off of one another as they both gain greater precision and breadth.

To be clear, TFA is about the second example of similar results under different conditions. Now they’d be looking for a third.

True. But all too often, people stop checking when their results agree with prior work. But then, eventually, someone is confident enough to publish results that aren't consistent with prior work.

Jeng (2005) A selected history of expectation bias in physics <https://arxiv.org/pdf/physics/0508199.pdf>

Also see Feynman's comment about the electron charge: https://en.wikipedia.org/wiki/Oil_drop_experiment#Millikan.2...

Lots of reasons. Statistical fluctuation. Equipment malfunction. Misunderstanding of setup. Underestimation of systematic errors. Improper data analysis. Sometimes, new effects that were previously unknown but don't involve new physics.

I once asked a senior physicist what to think about a 4 sigma anomaly that looked like it couldn't just be a systematic effect. He instantly replied "then it's probably two systematic effects pointing in the same direction".

Where can I read more about these hints?

You can go straight to review articles, which are generally available for free on the arXiv. Or if you want the firehose, you can scan for papers whose abstracts contain words like 'hint', 'anomaly', 'excess', 'discrepancy', and so on [0].

0: https://arxiv.org/search/advanced?advanced=&terms-0-operator...

I've provided a link to a blog that goes in depth on the previous results in another comment

https://news.ycombinator.com/item?id=21617254

https://en.wikipedia.org/wiki/Axis_of_evil_(cosmology)

Reply on news.ycombinator.com