A Better Crystal Ball?
foreignaffairs.com
foreignaffairs.com
This has never happened (a skirmish, let alone deaths) so how can we think that the forecasters are well-calibrated to predict the likelihood of this event? Because they predicted other, different, stuff?
In fact, yes. Prediction is a skill, skills can be developed and different predictors will have varying levels of skillfulness.
The biggest contributor IIRC is Fermi-ization: the skill of decomposing big hard predictions into many smaller predictions.
For example: what are the components of the question? They might include:
- The likelihood of deaths due to combat action. That can in turn include predictions of the relative effects of ship armour, weapons effectiveness, preparedness for combat conditions and so on.
- The number of people involved. If a skirmish involves a hundred ships on each side, 2 deaths is a very low estimate. How many ships are in the area? How many could reach it by December 31st? How many would be likely to be sent?
- The causes of conflict. What are the triggering events, and how likely are they to occur?
I'm sure smarter and better-qualified folks than me can argue all the details, but that's the point. You don't accept the question as a coin-flip with no priors. You go prior-hunting.
The article seems to be suggesting that this can be accurately estimated, but I feel like this would be a stronger claim if they cited something that was predicted and actually did happen...
Perhaps I misinterpreted the paragraph?
In this case, by the time DATE Y takes place, you have a clearly verifiable binary outcome, which you can use to judge whether the forecasters were correct or not.
There is some art to this -- the "event" in question needs to be very well-defined... I think the example they're using is a bit odd and they have much better examples in their book. The example of a counterfactual event (i.e., 'if X happens then estimate the likelihood of Y happening by DATE Z') adds a great deal of confusion here because you're predicting conditional probabilities. Still possible to verify but to your point, much more difficult.
At this point, you're asking forecasters to build a probabilistic graph and estimate causality, which is becoming more of a science (e.g., Judea Pearle's work) but still lots to do in that space. Anyway, I'm digressing.
So if I predict "aliens will land on or before December 1st" with 100% and no aliens appear, then my Brier score is 1 (the worst). If I say 0% confidence, then my score is 0 (the best).
There are dozens and dozens of ways of scoring prediction ability, as it happens: https://www.cawcr.gov.au/projects/verification/verif_web_pag...
Unless the prediction is "this won't happen" and then it continuing to not happen makes the predictor look accurate... (slightly joking here)
The article seems to claim that a specific event, which has never happened, can be estimated accurately. Is that really possible? How can the accuracy be assessed?
Suppose you make predictions about 100 events, and only 25 of these events actually take place. Your baseline might be in relation to these 25 events, and your 'superforecaster' needs to be better than an existing model, which could be really bad at predicting these odd/weird outcomes. Now you're better than everything else! :)
No, that's not right. The general idea is that you have forecasters who make predictions on multiple topics A, B, C, and D that have all not happened at the time of the prediction. Then, by looking at the outcomes of A, B, and C you can judge the ability of the forecaster to make prediction on topics what have not yet occurred at the time of prediction. You can then become confident on their ability to predict topic D before it occurs.
The argument you're making is the classic skeptical argument about the philosophical problem of induction: just because some procedure (e.g., an astronomical model of the solar system) has made successful predictions in the past, we cannot rule out the possibility that the sun will not rise tomorrow.
You can see where the problem with your argument lies by noticing that it depends heavily on the definition of what a "topic" is. If we have made successful predictions in the past on (say) the tides, and I make new predictions in the future, one can always point to some feature of the new predictions that could arguably make it a different topic, e.g., maybe El Nino is this year, or maybe the tides were for a slightly different region, etc. New predictions always differ from previous predictions in some way, but this does not mean the predictions are useless.
The resolution is that "topic" is not a precise idea. Rather, if a forecaster is successful on topics A, B, and C that are all in a natural reference class, and D is also in that reference class, then this is evidence — not incontrovertible evidence, but strong compelling evidence —that they are likely to be accurate on D too.
Thanks, that really clarified the idea for me :)
I'm not saying this event can't or won't happen, I'm questioning the claim (maybe I'm misinterpreting the article?) that we can accurately forecast the likelihood of it happening, even though it's never happened.
There was much point scoring done after the 2016 U.S. election about polling and predictions being wrong, particularly against Fivethirtyeight, but where I think critics got it wrong was, when someone says there's a %95 chance of an event going in one way, it's not valuable as a prediction, but it certainly is as an accurate assessment of the impact of the results. In 2016, if you told people that the %95 implied probability of winning translated to what felt like paying out a 19/1 loss, a lot of people would have agreed with you.
You can reliably bet on the effect of the upset being inversely proportional to the predicted probabilities. When I see election predictions that are 60:40 I think the 2nd order impacts of the result will be slightly lopsided, but within the realms of expectations. In this sense, %54 chance of winning close races may be better for stability than perceived polarized %95 ones because of how people perceive the results of a loss.
> You can reliably bet on the effect of the upset being inversely proportional to the predicted probabilities.
Broadly, yes, if you think of it in an information-theoretic sense. Rare signals carry more information than common ones.
You would think with that much planning we would have paid attention to experts decades ago warning that we needed more charged fiber (N95) in storage in case of a pandemic at a cost of ~0.0001 trillion ...
P.S. I have been downvoted for saying this.
I do agree that keeping this high level and abstract (scenario planning + probabilistic estimates) isn’t necessarily doing them any favours. Their book on the topic is better, and I do appreciate their argument that it’s not about being a perfect predictor of the future, but simply better than what we have today.
What you can do, however, is control your exposure to outcomes of unpredictable events. You don't know when a pandemic will hit, but you know it is is matter of time, just like getting hacked. So prepare and design accordingly. Simple. No need for this superforecasting nonsense which doesn't even work.
Other events are not.
That’s part of the beauty of the problem. You need to be able to pick the right events to predict and ensure the event is well-defined enough to actually test. Your own examples of why this wouldn’t work are exactly what they discuss as the problem... Eventually a lot of forecasts are correct; what governments, companies, and leaders need are forecasts that are time bound, as that’s how they plan.
I recall, but cannot locate, that questions related to the outbreak of trans-species respiratory illnesses from China have been canvassed in the past. It does appear that the US intelligence community expected a pandemic to emerge: https://www.mercurynews.com/2020/03/16/analysis-yes-people-d...
Tetlock has data piled on data.
https://www.nationalgeographic.com/science/2020/04/experts-w...
I'll bite, what?
I know he's rich and successful, so he has deeper insights than most.
But it doesn't mean he teaches them, as far I can see he just uses them. For instance he's very good at doublespeak.
You may not owe authors whom you feel are charlatans better, but you owe this community better if you're posting to it.
After a great deal of crunching and study he said they did come up with a model that could do it reasonably well. Then realised it could be replaced with a simple rule:
Are there a bunch of numbers at the top? Good news.
Are the numbers buried way down? Bad news.
Another simple rule, by the way, is timing vs executive compensation events. If there's an announcement just before a big block of options or RSUs are vesting, then it's good likely to be good news. If it's being posted far away from vesting dates, then it's likely to be bad news.