Hacker News Reader

Comment by hardtke | Hacker News Reader

The most important lesson I learned in graduate school came from a scientist at Los Alamos who told me "three sigma effects happen in physics much more often than they should"

View on HN

You think that's bad?

How about in finance, where the CFO of Goldman Sachs bemoaned that they had "25-sigma events, several days in a row" during the financial crisis.

http://lukenotricks.blogspot.com/2010/03/fabled-25-sigma-eve...

Surely at the claimed 25-sigma point, the problem is more with our ability to measure than the probability itself? Claims about the space shuttle's reliability come to mind.

Am I the only one who feels that a claim of probability on this level is basically religion on one level or another?

The problem is that the model was fantastically far off from reality. (For whatever reason - incompetence, bad inputs, willful deception, religion - doesn't matter for this discussion.)

I could construct a prediction that the high air temperature in New York tomorrow will be 80 degrees Celsius with a standard deviation of 3 degrees. Then when the observed value is actually 5 degrees, that's 25 standard deviations off from my "prediction". But it doesn't mean that the observed event was actually a 1-in-billions occurrence, that New York experienced an astronomically unlikely cold snap. It means my prediction was unrealistic and wrong, that it didn't model reality.

The math works, based on the model. The model works based on the human guessing what the model should be. The real problem is that '25-sigma' events in the model are completely overshadowed by the not-all-that-small chance that the model is wrong. So talking about 25-sigma events as if it was something you could manage and control, when they are so far below the noise level of human error (or lack of information/understanding) is nonsense. It tells you something, but if you throw away the context, it will happily lie to you.

Computers can measure to arbitrary precision, so that's not the problem, and if you have good reason to think that you have 25 sigmas worth of evidence, that's still very different from a religious belief which needs infinitely more evidence to justify, or a religious belief of having unshakable meta-belief in the probabilistic model such that you never update it when it's wrong. The problem is really just a bad model, or (as with Challenger) ignoring the computer/engineer/model completely. Probability is in the mind; "events" happen, not "25-sigma events"; if your model is predicting a very low chance of something that happens much more than warranted, your model is very wrong. Is it so wrong it needs to be cast out entirely? Possibly. I'm more interested in how it gets updated when it's wrong.

Certainly. When reality disagrees with your theory you don't blame reality, you blame your theory. The idea that there would be yet another housing bubble which popped and that this might cause a massive number of defaults from "sub-prime" lenders is not exactly fucking rocket science, it should have been on the mind of anyone and everyone investing in that business. That's not a 25-sigma event, that's a 0-sigma event, it's predictable and inevitable. By making it seem like the inevitable was impossible these charlatans transformed an ordinary housing bubble into an existential crisis for the world financial industry. It's a shame more of them aren't in jail.

Reply on news.ycombinator.com