How Good Are FiveThirtyEight Forecasts? | Hacker News Reader

How Good Are FiveThirtyEight Forecasts?

projects.fivethirtyeight.com

34 pointsjacobedawson48 comments

All of the people complaining about how Five Thirty Eight is a fraud because they "predicted" that Hillary Clinton would win should read this. They gave Trump a significant chance of winning, more than almost any other poll aggregator did. In fact, before the election, Nate made a post outlining all the reasons that Trump could win and gave him a 36% chance of doing so. They literally said that trump would win based on polls like that 36% of the time, so it shouldn't have been that shocking that he did.

https://fivethirtyeight.com/features/election-update-dont-ig...

You might say that a forecast that's so ambivalent isn't useful, but if someone told you that there's a 36% chance of rain, you'd plan your beach trip accordingly. I don't see why predicting a 36% chance of the underdog winning the presidential race is any less informative.

can downvoters elaborate on why they disagree?

It's probably a confusing the map with the territory sort of thing. FiveThirtyEight predicted Clinton would probably win and didn't. Many people are (understandably) upset she didn't win and they're lashing out at FiveThirtyEight for having given them what turned out to be false hope.

As evidenced slightly in this thread already, my observation is that most of the critics actually wanted Trump to win and are gloating that he did.

It might also be something a bit different:

There's a lot of people on the left who are angry with Clinton and the Democrat establishment for running a very technical/data-driven campaign that turned out to be very flawed and ran counter to common sense (even her husband was telling her to campaign in areas she was ignoring due to this "data"). The technically driven 538 sort of represents that attitude.

Nate Silver has also started to stray into political punditry which is turning a lot of people off. People who liked him for his data are now having to hear where he stands on issues.

Mixing personas is always challenging. See also people who want to be both entertainers and journalists depending upon what suits them at a given moment. Of course, there's no reason someone can't be both a pundit and a data scientist but one casts suspicions of lack of objectivity (which, of course, never totally exists in real life) on the other.

For me, I’ll always monitor and follow his poll analysis. But something about how he handled 2016 soured my view of him. It’s like he didn’t just “predict.” He insisted on being correct in a way that was too overt.

Further, let’s not forget all of the dirty deeds orchestrated through Facebook, online, and hacking. The election in three key states swung by tiny amounts. It’s not correct to blame what otherwise worked for the perfect storm. In fact, I don’t remember 538 sayin anything about fake news or election hacking prior to the election. It was all polls of polls.

538 gave Trump and the Cavaliers similar odds in 2016, something I pointed out to a bunch of people after the Cav's won but before Trump did. Golden State were substantial favorites, yet the Cav's beat them. Nobody is blaming 538 for calling GSW the favorites, few people think their probabilities were wrong. Why is it so different for the presidential election?

This 'calibration' is a missing link for machine learning.

A prediction is much more valuable if you know how good of a prediction it is than if you don't.

I'd caution about putting too much stock in calibration, though. It's worth noting that you can achieve perfect calibration just by giving each of n candidates 1/n probability of winning.

I don't think you can. If you give every candidate a 1/n probability in winning, but the actual true probability is less than 1% (I'm looking at you, Vermin Supreme[0]), then your calibration is going to be way off. Good calibration is having your 65% chance of victory be right 65% of the time.

[0]https://en.wikipedia.org/wiki/Vermin_Supreme

Right, but say in 2016 you gave a 1/3 chance to Trump, 1/3 chance to Clinton, and 1/3 chance to Vermin Supreme. Exactly one of them will win, so your calibration for 1/3 will be perfect. If you do the same for elections with different numbers candidates, you can have good calibration overall without actually producing any information over the base rate.

I disagree. Suppose you have an election with 3 candidates that runs every year. If you predict candidate A win at 33% confidence, but that candidate gets elected every year, your calibration will be way off. Calibration is about aggregating, so you can't really cheat it. Your 33% prediction has to be right 33% of the time - it's really hard to fake that.

Maybe I'm misunderstanding your position, though.

That's right if you only predict the probability of candidate A, but if you are predicting 33% for all three candidates each year, your 33% prediction will consistently be right 1/3 of the time in aggregate.

This is good in the sense that having poorly calibrated forecasts is bad, but it's not quite as strong as you might want it to be. Imagine that you're tasked with predicting the next state of a traffic light that just cycles through green, yellow and red over and over again. If at every step you forecast that there's a 33% chance of each color, you have a very well-calibrated forecaster, but it's clear that you can do better.

FiveThirtyEight probably actually is doing better, but I suspect that they're not choosing to highlight those stronger metrics because they're complicated to explain.

But isn't that exactly what the last third of the article discusses?

Can anyone explain how 538 got the last election so wrong?

Its quite obvious in hindsight their model is flawed, but Im not understanding how is it flawed.

Did they? As far as I can recall 538 has always been clear that they deal in probabilities, not binary predictions. They gave him a small chance of winning for a reason, and it turned out chaos favored one of the low odds scenarios.

If you're referring to either the midterms or the 2016 presidential election, they didn't get either particularly wrong.

This gets to the core of Taleb's beef with FiveThirtyEight - what political forecasts are calibrated? The ones that were forecasted 6 months out, or the last forecast before the end of the election?

They may only be calibrated when enough of the facts are in to make predicting easier. Still useful, but not calibrated in the way the general public might think.

If you dive a bit more they look at each forcast at each time interval for which it is made. So they include the 12 month out one, the 11 month out one, etc when assessing calibration.

Ah, that's very useful to know. I take my statement back then, this is better calibration than I thought.

They obviously got 16 very wrong. However, what is even more concerning is that Nate lately became a political hack, doing the ugly work for the democratic establishment. Just look at his twitter you can clearly see it. So I would take any "data science" he does with a grain of salt.

Edit: An example of his tweets:

"It's probably worth noting that while this group, Justice Democrats, calls Biden "out-of-touch" with the "center of energy" in the Democratic Party, only 26 of the 79 candidates it endorsed last year won their primaries, and only 7 of those went on to win the general election."

Why didn’t he add the statistic that Biden already ran 2 times in the past and won 0 times.

I have seen his Twitter feed, do you have any specific examples? Obviously he's talking a lot about Democrats right now more than Republicans because Trump is the presumed Republican nominee, but covering the race isn't "doing ugly work for the Democratic establishment."

Please explain to me what is the point of such a tweet.

It contextualizes a remark from a group that claims to speak for the party's electorate by providing hard numbers. That's what people who follow Nate expect from him on Twitter.

Edit: to be fair, if memory serves JD mostly/entirely endorsed primary challenges rather than incumbents. The base rate for primary challengers is pretty low, so ~1/3 is a respectable win rate and I would concede that it's a little disingenuous not to mention that.

The group doesn't claim to speak for the part’s electorate. It claims to speak for a particular ideological faction, toward which it also claims the party electorate has moved since the last election, so that a candidate representing the faction that was uncontroversially still dominant in 2016 is now out of line with the dominant view in the party.

Nate seems to be implicitly arguing out that that claim would not have been inaccurate if made in 2016 (and further that the claim that may have been implied but was not made that the same faction is positioned for success in the general election should it succeed in the party nominating contests would also.have been untrue if made in 2016.) But that is silly, since it's a claim about how the 2020 landscape is difft than 2016.

Or, maybe Nate is conflating endorsements with predictions rather than advocacy, and trying to argue that the rate of success of the groups endorsements is somehow indicative of the reliability of their descriptive statements.

Or maybe Nate, who aside from running a prediction site is also an activist aligned with the faction being criticized against the one doing the criticism, is not posing a rational argument at all, just trying to provoke an emotional response by posting something that vaguely insinuates that the criticizers are wrong in some way.

Eloquent writing definitely helps explain my point better. Thanks.

This does not contextualize anything in any way. He is selectively picking data to form a narrative. I dont think you are objective at all so I will stop trying to convince you but lastly think about this:

Why doesnt Nate reply to Neera Tanden, CAP or Nancy Pelosi with “stats” on medicare for all support which is polling at 72%?

> I dont think you are objective

If you're implying I'm pro-Biden, for the record, he isn't even in my top three. I don't read Nate because he supports my beliefs, I read him because I think he has a good-faith interest in accurate predictions (even if it is impossible for a human being to be 100% objective)

That statistically the fringe wing's candidates don't win?

Fringe!!! You would name Bernie and AOC and politicians with similiar ideology fringe candidates? This is trying to form a narrative by nitpicking data.

He forecasts elections, and his models include endorsements. This tweet discusses a group that issues endorsements, their stance on a candidate who just entered a primary, and the win rate of candidates endorsed by said group. This is essentially a perfect tweet illustrating something that a forecaster would do.

I still expect him to tweet to Biden saying he ran 2 times and won 0 times.

The fact that he hasn't tweeted about every possible statistic is irrelevant to your ask about "the point" of the referenced tweet.

Aahh still going over your head. The whole point I am trying to make is that anybody who is fully objective and data driven would refrain from making such statements online, especially when it is uncalled for. Nate has been consistently taking a pro-establishment stance even when he was proven wrong over and over again. To say it in another way he could have said: Justice Democrats, founded in 2017, went from having zero members in the House to 7 within a year. Including AOC, one of the most popular politicians in the US.

It's not going over my head. You support Justice Democrats.

They did not, he has not, and one should not any more than one always should.

538's forecast was easily the most accurate for November 2016, by a decent margin. The majority of critics of 538 are "sore winners" who delight in their side having won despite being given a less than 50% chance, but as Silver pointed out at the time: his chances were similar to those in Russian Roulette, and most people consider that too risky to play.

Right. It's like saying "Ha! You were wrong! You said there was a 83% chance I wouldn't roll a six, and I did"

More like saying, "We said there was a 17% chance that it could go the other way, so no matter what happens, we aren't wrong."

That's why it's important to do what 538 did here. Look at all the different places where they made 5 in 6 predictions ( and there are a lot of different races) and make sure that 5 in 6 of them happened.

This is disingenuous. A prediction is a prediction. There is a world of difference between 51% certain and 99% certain. People excited that their data science models get good predictions on their test set without considering, say, the RoC curve are setting themselves up for failure.

If a model says 75% chance of X rather than Y, that doesn’t mean you should expect X. Depending on context, you might be better off flagging it as unclear or even safer to conclude a possible Y.

Testing calibration of many models across many different trials as a test of data scientist competence is a little strange, but if you want to answer the question of how valuable the predicted likelihood’s are, it’s a decent measure.

538's forecast was easily the most accurate for November 2016, by a decent margin.

No, Investor’s Business Daily was the most accurate.

If thats your understanding of making a prediction then he was accurate by your standards. You clearly did not watch the election night predictions. NYT live model was miles better than 538s.

https://fivethirtyeight.com/features/how-i-acted-like-a-pund...

Your link is from 2016 before the presidential election but after the primaries. Nate is admitting that he underestimated Trump in the primaries and talks about what happened. This does not seem particularly relevant to the subject of election night predictions.

This is regarding his bias. If anything the link is even more relevant. He repeated the similiar mistakes.

His main mistake as highlighted in that article is that they didn't use statistics for the nomination process, but guesswork. For the election proper, they did use statistics. So no, they did not repeat the mistake at all.

His election night model wasn't good, but he isn't known and respected for his election night model, so who cares?

Reply on news.ycombinator.com