Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast
statmodeling.stat.columbia.edu
statmodeling.stat.columbia.edu
Presidential assassination, war, video proof of something incredibly heinous (pedophilia?), etc. can absolutely lead to these outcomes. You don't even have to go that far back. Nixon and Reagan flipped states like no-one's business.
I do however agree, that 538's state-state correlation model seems weak.
California and Alabama would only flip during a wave, and that wave would consume any and all states. The fact that 538's model doesn't strongly show that pattern is a failing of it. But, it is not clear if a model that inaccurately models the unlikeliest of events (california flipping while Florida stays blue), does not necessarily mean that it is terrible predictor of it's primary target (Presidential likelihoods).
As a data scientist, I can totally understand Nate's hesitation. Do you impose strong priors on the model to reflect strong domain intuition or do build a model that best characterizes the data it is based on. In the presence of infinite data, you should abandon all domain based priors. For single digit data points, priors are essential. For any number of data in between, it is anyone's best guess.
As to whether it's sarcasm, I'd describe it more as an inside joke with just a touch of the self-aware intellectual arrogance Physicists are famous for (see Lord Rutherford's "All science is either Physics or stamp collecting").
Trying to explain a joke is always dangerous so hopefully what follows won't be a mistake, but here goes:
The way I have always interpreted the first sentence is to hear Fermi saying that when you are doing an experiment, you should have a deep understanding of the family of curves or behaviors the system under test is expected to follow, including the full range of curves that would follow from interactions that are wildly different from what you might naively expect. If you really understand the Physics of the system (this is where it starts to blend the line from advice to a joke), the measurement of one single data point should be enough to tell you which of those possible curves describes the actual behavior of the system (the joke being both that it's funny to be that arrogant and that it's obvious to anyone you'd tell this joke to that mathematically you need at least two points to determine a line, so by saying one point gives you not just a line but a curve the speaker is purposely going over the top for fun). Moving into the second sentence ("two points gives you the distribution about the curve") takes the statement into full-on joke mode, with the comment shifting from an observation about knowing your Physics to an insider dig at the relationship between theoretical Physicists and experimental Physicists (Fermi being one of the greatest theoretical Physicists of all time). When he says that two points gives you a distribution about the curve, he's saying he as a theoretical Physicist understands the underlying Physics of the system better than the experimentalist understands the noise in their experimental hardware, or alternatively that the noise in the hardware is sufficiently uninteresting as to be irrelevant to him. The former view would simply be arrogance but leaving the second option open circles the joke back to include a bit of insider self deprecating humor in that he's purposely ignoring experimental error, a thing theoretical Physicists are famous for doing.
It's bizarre there was no better analytical/computational way to come to what they were expecting.
Something could flip California and Alabama (example, Trump starts defending Roe v. Wade and in response Biden somehow manages to sound like he's opposing it). This would probably be some latent hidden variable, like whether the candidates are seen as socially conservative, which would effect all states (though California and Alabama would be the most impacted).
Constraining against them won't improve your models fit (usually by definition), and it doesn't always improve robustness (at least for situations near average)-- because they're acting to debias the model in ways that you otherwise don't have enough degrees of freedom to address.
A negative correlation here is also potentially historically supported, in the sense that sometimes DEM/GOP candidates are philosophically reversed in some way relevant to the state. As in, "The only way a GOP would get elected in X is if they had the DEM position on subject Y which would make them lose state Z, who cares as much about that subject as X but in the opposite direction."
Now-- it doesn't seem likely case in this election (e.g. Trump is not (currently) a massively pro-choice republican), so it probably shouldn't apply here-- but it's isn't hard for me to imagine how a negative correlation might show up out of the historical data.
The Economist model does exactly that, and all of their correlations are positive.
I recommend reading their methodology, they know what they're doing (I wouldn't say the same about 538). Andrew Gelman has developed some of the Bayesian methods and software that people like Nate Silver use, he's the main author of what's considered a reference book on Bayesian statistics.
It's clear that the models are tuned differently, but from Silver's replies in the PS's, it seems that he's ok with these artifacts being part of the model.
Just because intuition says it should be longer odds for Trump doesn't mean that's right.
It's very interesting to see how long it takes people to do things. I am amazed that entire article took 1 hour to type up. I've spent entire afternoons trying to write shallower pieces of work.
Granted, i could just be misreading this post. :)
https://projects.economist.com/us-2020-forecast/president
You can compare this to the 538 model and see where these two teams and forecasts disagree.
The model says that Trump has a 1 in 10 chance of winning. With a fair 10-sided die it makes sense that you have a 1 in 10 chance of any given side rolling face up. But what is the die that is being rolled in these election statistics? What is the "chance" element that is being predicted?
In the dice toss scenario, we know everything relevant. In the election scenario, we don't.
A model like this is attempting to say "these are the rules we think exist. Based on the rules, and assuming the data is off by some random distribution, here's what we think could happen".
What different forecasters disagree about is what the rules are. For example, the relevance of certain demographic characteristics and the potential variance between polling (conducted prior to the election) and actual election results.
There's a huge amount of assumptions, and forecasters disagree on those assumptions. We have very little historical data (polling is very recent) and even with complete historical data, future elections do not always conform to past elections.
The Democratic Party proved it was not as progressive as they thought as Sanders lost the primary. The reality is, the country as a whole is also not as liberal either, regardless of what these pollsters are asking people. You think the party is youthful, and ready for progressive ideas, but alas, the party wholly rejected an amazingly progressive candidate in Sanders. You think everyone’s super pissed at Coronavirus handling, and police brutality, healthcare, but alas, you find out people associate BLM protests with crime, and the virus with China, and socialism with unfair wealth redistribution. We can keep learning this the hard way I guess, this is America after all.
It’s important the technical discussions are happening this time around, because there was virtually none the last time. The post mortems for these forecasts being wrong again should be a death knell for accumulating bad data. I’m certain the models are good, but I’m not certain the data is.
Anyway, if you want my hot take, the conditional forecasting is to save their ass on election night from being embarrassingly wrong again. Imagine writing a giant if-statement that looked something like ‘and if(imWrong) changeMyAnswer’.
Well Nate Silver wrote a full critically acclaimed book about why these types of forecast are more useful (and accurate) in reality because they account for uncertainty - he has been doing this for years, ever since he used to write similar algorithms to help bookies pick odds for sporting events, so I think your hot take isn’t based in any world of facts or knowledge on this.
Don’t trust a forecaster that says with certainty that a certain candidate will win, unless they have also bet their life’s earnings on it. Showing your statistical confidence level isn’t a bad thing.
If you listen to what they say, they admit they were not able to measure for the no-colllege male demographic in 2016, or in other words, they couldn’t model identity politics. Why couldn’t they do that? I’m not sure, but they are certain they can this time around because they saw the 2016 data and now believe they have more complete data to not make the same mistake again.
They are looking at elections as if there are hundreds of millions of elections that happen every day and the data speaks for itself. No sorry, there’s very few elections to extrapolate the way they are doing it, and you really need to do sociopolitical analysis of things like a demographic identity bloc (no-college whites that feel some way about things) that really get you the accurate undercurrents that can sway an election.
Lastly, it doesn’t take a genius to sit there at 10pm on election night and go ‘well if Florida and Michigan went this way, then probably so will these other states in flux’. ‘Our forecast becomes more accurate as we get the actual poll closing numbers on election night’, ah I see, you’re all geniuses, I should have known.
Anyways, we’ll know soon enough.
You seem to have a fundamental misunderstanding of what FiveThirtyEight is trying to model, versus what pollsters are trying to model with the numbers they publish that FiveThirtyEight consumes. The kind of demographic weighting you're complaining about FiveThirtyEight being bad at is something the pollsters do, and is outside the scope of FiveThirtyEight's forecasting models.
I think you possibly misunderstand what 538 _do_ a bit. Their data is based on polling, so they can only work on what the pollsters do. Historically, pollsters didn't pay that much attention to education, beyond using income or class as a proxy for it; one middle-class white man was pretty much like another. This worked quite well historically, but no longer does (and it's not just a US phenomenon; it was also a contributor to polling problems for Brexit, notably).
In their current model, 538 assume a higher rate of uncertainty than last time round; also, some pollsters now model education. But really there's not that much they can do about stuff that pollsters don't ask about.
I’ve got some basketball statistics to populate 538s model if their interested. Lebron did pretty good this season, hopefully they can correlate that with the black vote.
Their model is not transparent on any level, because if they make it transparent, we’d easily be able to see why it’s ridiculous.
What does missing the mark mean though? In 2016 they proposed a c30% chance that Donald would win, and a 70% chance Hillary would win. Does that mean they were wrong? Not really, because that's how probabilistic forecasting works - and they stated their confidence interval - they were 70% confident that Hillary would win, but thought there was a 30% chance Donald would win.
The FiveThirtyEight forecast for the Democratic primary [1] gave Biden the highest chance of winning for most of the process. He did have a steep drop in the month before Super Tuesday (followed by an equally steep rebound), but still, I wouldn't say the forecast was especially bad. That said, polling is always worse for primaries than general elections, since there are more candidates and fewer voters.
[1] https://projects.fivethirtyeight.com/2020-primary-forecast/
Once again, I’m not an expert, so I recommend looking for additional explanations, if you’re interested.
A program which randomly generates outcomes for each state, based on probability distributions inferred from the polls, and calculates who wins the election given those outcomes. They run the program repeatedly and report the proportion of simulated wins as the probability of winning. https://en.wikipedia.org/wiki/Monte_Carlo_method
So, think of it as saying these facts basically describe a ten sided die. With no other knowledge, the best you have is that you expect it to behave the same as any other ten sided die.
(There are a couple of caveats about election forecasting as opposed to weather forecasting. The first is the "October surprise," a sudden revelation that changes the election. This cycle, it was arguably Trump's covid diagnosis, although that tended if anything to push the results further in the direction they seemed to be going on their own, rather than upset any trend. The second is that, unlike with weather systems, measuring voter behavior (and widespread reporting on these measures) can change people's behavior. The effect of this is hotly contested, but one of the many explanations of Trump's victory in 2016 which hinged on turnout in a few key states is that those states were predicted wins for Clinton, so Clinton voters didn't bother voting. Despite occasional jokes to the contrary, it doesn't rain just to spite the weatherman.)
The frequentist interpretation is roughly that if I go around making my best possible predictions, and we lump together all the things that I predict at 10%, about 1 in 10 of those things happen and the rest don't. But I wouldn't be able to be more specific about which ones in that group are more likely than others.
The Bayesian interpretation is that I can really view the world as flipping coins -- I don't care whether it's due to my lack of knowledge or "true" randomness -- and as far as I can tell, the coin flip involved here is 1 in 10.
We can also use a gambling interpretation. Here's one based on security of python's random module. Imagine the following three lotteries I offer you. In lottery A, you get $100 if Trump is elected. In lottery B, you get $100 if the following python code returns true on my laptop:
random.random() <= 0.09999
random.random() <= 0.10001
Now there's an interesting extra layer to all of this because it's a model predicting, not a person. In a short space, I would basically say that we've trained models to predict in ways that are not inconsistent with any of the interpretations above, when put into situations where that is testable. Then we use them in situations where it might not be, like this.
IIRC, 538s election models (house, Senate, and Presidential) have been quite accurate in aggregate.
Andrew Gelman (the author of this post) has also done a bunch of work on how different parties supporters become more/less likely to respond to polls based on what the current results are, which has been incorporated into the newer forecasts.
It's different than probability (1/10 = .1)
(I suspect that counter-pedantry on these lines might be part of why your post is getting downvoted; I wasn't one of the downvoters fwiw.)
The probability of an electoral-college tie is <1%
At the bottom of the 538 page it says, " If you choose enough unlikely outcomes, we’ll eventually wind up with so few simulations remaining that we can’t produce accurate results. When that happens, we go back to our full set of simulations and run a series of regressions to see how your scenario might look if it turned up more often."
I interpret that as running a regression (linear?) and extrapolating it out to the tail where the conditioning is happening. This should eliminate the issue Andrew is seeing?
In that particular WA-MS example, if Trump suddenly took more liberal positions and somehow won WA (e.g., announces he's pro abortion), he would in fact be more at risk of losing Mississippi. The idea that these two states are in play already is fringe and would require some major idealogical (or other third variable) shifts.
The negative correlations don’t make sense. Maybe it’s a small problem and the model is solid overall, but... I don’t think you can justify that one effect.
Specifically, that when you get off into the weird situations like Trump winning Washington state, it's likely something incredibly weird has happened - something that likely has no historical precedent, so it may actually be a more sane thing to do to assume that now almost everything is backwards and Biden would win a bunch of states he shouldn't either.
To me, this points to a general willingness in the 538 model to just go "who knows" and build in some room for insane things to happen on the fringes. The Friday podcast episode about the 538 model specifically mentions that they have large/fat tails on their distribution that make it nearly impossible for someone to get over 95% chances of winning on a national level, and these sorts of wild results seem like the outcome of that. If you bake in an assumption that there's always a 5% chance of something crazy happening, that chance has to come from something in the data somewhere that reflects the ability of that to happen numerically, and thus will have numerical outcomes that seem impossible.
The negative correlation makes sense when we think about how difficult it is for everyone in Washington to suddenly turn conservative and everyone in Mississippi to turn liberal. Much more likely is that the crazy thing is that the candidate or circumstances changed in some way.
It makes more sense if we ask...if a candidate wins NJ what is the chance they also won AK?
It's a neat possibility to think about though. If there were enough people who did that, it would really depend on the demographics moving and where they're going. It could swing the election either way. I wonder if anyone has found numbers on this and attempted to model it.
Moving out doesn't stop you from voting. I didn't change my voter registration when I moved from San Francisco to China. Years later, back in California, I voted in San Francisco, where I was still registered, despite residing in Hayward.
For verification purposes, they asked me when I voted what my address was. I was allowed to vote despite not knowing my own apartment number.
I don't have a methodology, because I'm not a pollster with dozens of people at my disposal. I am just bemused and annoyed that things like 538 continue to be taken seriously when they continue to ignore sociological, historical, and cultural factors in favor of an overly-complex quantitative model.
Re: the upcoming election. I don't think we can be sure, yet. Certainly it will be close, and the Biden at 90% to win estimations make no sense to me. Biden is a much weaker candidate than Hillary and he continues to make blunders (i.e. I guarantee that his comments on fracking in the last debate just lost him Pennsylvania.) Trump seems to be finding a lot of allies in strange places, e.g. African-American celebrities. That may be an isolated incident, or it may signal some big unexpected changes.
At this point my estimation is Trump-Biden 55-45, for the simple reason that people tend to vote for economic issues and Trump has a better "perception" on this issue. "It's the economy, stupid." as James Carville put it.
Alternatively it might be because you're objecting to the results of a well-documented statistical process, and then when being questioned saying things like "I don't have a methodology".
You'll notice that the comment you replied to (saying it was similar to your point) is not downvoted into oblivion.
That’s your view; however, he is polling much better than Clinton, which would indicate that voters don’t necessarily agree with you (or else just that peoples’ opinions of Trump are lower than last time round, or a combination. But really it hardly matters).
> (i.e. I guarantee that his comments on fracking in the last debate just lost him Pennsylvania.)
Looks like 20-50,000 people employed in fracking plus industries supported by it in Pennsylvania. And presumably most of those would be voting for Trump anyway; it’s not like Biden’s views on fracking were a total black box til now. So it only really matters if it’s very close anyway.
>At this point my estimation is Trump-Biden 55-45, for the simple reason that people tend to vote for economic issues and Trump has a better "perception" on this issue. "It's the economy, stupid." as James Carville put it.
Indeed, Gallup reports that 56% of Americans believe that they are better off than four years ago (https://news.gallup.com/opinion/gallup/321650/gallup-electio...).
My model is simpler. If Trump wins every other state he won in 2016, he only has to win one of MI, PA, WI, MN, or NH/NV. The first three he won in 2016 (and, as you say, Biden's views on fracking may very well cost him the state); MN Trump lost by 1.5%, so the state is only sightly behind the rest of the Midwest bar IL, and half of Minneapolis being torched this summer probably pushed the state over.
If you want to know where the models are going to break down it is more likely that the state-level polling has over-corrected for the factors that caused them to miss the swing to Trump in 2016 and Biden's numbers are even better than what polls are saying (a prediction based on looking at polling at the congressional district level and then seeing how that differs from state-level polling -- the numbers are the district level are closer to national numbers than the lower statewide numbers for swing states.)
Nate Silver authored an adjustment to polls used in that model. Polls have more impact if they are more representative of statewide turnout among demographic things he chose like “black” and “low income.” This is why his predictions were so accurate for Obama’s 2008 and 2012 elections, and likely why they were so inaccurate in 2016.
Gelman’s own grad student is the only person to have academically published this approach, in a paper about polling Xbox Live users.
These guys sort of make a thing that is the same in many more ways than it is different. Why not just share the code is the biggest question?
The only sensible way to predict probabilities that aren't extreme is to tell people how the model works and the figures it is currently spitting out. That's is the great thing about these kinds of blog posts, people are kicking the tyres, not just looking at the car.
Nobody predicting a one-off election with a rather special candidate would summarize a 33% chance as equivalent to having no chance.
You're not wrong, but you should not do this.
Silver did a nice writeup of the whole experience: https://fivethirtyeight.com/features/the-real-story-of-2016/
The mistake in 2016, IMO was a) the extrapolation that came from those polls and b) people paying way too much attention to national polls, which have very little connection to electoral outcomes, given the electoral college.
Also perhaps c) the larger public not “getting” statistics in the way they’ve been presented. The NYT had, if I recall, Clinton at 90% chance of winning. That still means that in one of every ten flips of a coin is a Trump win. But people read “90% chance” as “definite win”. I don’t actually know what anyone should or could do about that.
Nobel Prize winner Daniel Kahneman's life's work is about this, what he calls "System 1" and "System 2" of our brain, where System 1 is a fast responder that provides insta-feedback but is largely incapable of processing mathematical inputs. His 2011 book "Thinking Fast and Slow" summarizes his work well.
I'm not sure popular media can be trained to frame statistical probabilities in a way that doesn't provide people with the certainty they crave. But who knows?
Say a national poll predicts 55% of votes for Clinton, 40% for Trump. Whereas 538 predicts 70% chance of winning for Clinton and 30% for Trump. It’s easy to confuse the two and think the second prediction is much better for Clinton when it might be much worse.
538 are aware of the problem, and combating it with a cartoon fox (and better visualisations).
https://projects.fivethirtyeight.com/2016-election-forecast/
28.6% is between 1/3 and 1/4, definitely not 1/5.
I find this argument strange, because black turnout was unusually high in 2008. That should have a negative impact on the accuracy of statistical adjustments, not a positive one.
However, we will never know, because they never published the code.
I think that kind of adjustment is usually the responsibility of the pollsters, with their likely voter models. I don't think FiveThirtyEight directly tries to also apply such an adjustment, because that would be at serious risk of overcorrecting.
Similarly, this year many pollsters have added level of education as a factor to their demographic weighting, to address a shortcoming in their 2016 performance. FiveThirtyEight consumes those poll numbers without adding their own layer of demographic adjustment.
I don't think it's likely but if those polls are indicative of what's actually happening, we're talking about potentially a 2-4 million vote swing in Trump's favor. Here's a link to estimates of voter turnout in 2016 [2].
[1] https://fivethirtyeight.com/features/trump-is-losing-ground-... [2] https://www.pewresearch.org/fact-tank/2017/05/12/black-voter...
That's just a really rough calculation and doesn't account for the Hispanic vote either in that article.
This would then mean that if 98% vote Trump or Biden in 2020, we'll see something like 84% of the black vote for Biden and 13% of the black vote for Trump. A 5% overall change using 2016 voter participation numbers is still somewhere around 700,000 vote change in the black vote, which is certainly not insignificant. Adding in the change in the Hispanic vote (a margin change from 37 points in 2016 to 23 this year), this could certainly change swing state outcomes.
There are plenty of reasons our predictions might not match reality, but they're not going to be wrong in that direction for that reason.
We’ve seen that variable before.
That's a valid intuition to have but you can also clearly make the argument that if Trump wins California you're in such a weird scenario that using the traditional wisdom about correlation is dangerous. The point that 538 have tried repeatedly to make is that firstly: if you're conservative in your level of confidence you'll give a higher likelihood to outliers, and secondly: It's not particularly useful to focus on whether X has a 3% or 4% chance.
If Trump wins California, we aren't going to be talking about whether the chance was 3% or 0.3% we're going to be talking about that Nuclear explosion that wiped out 25million Californians.
For the same logic the reason that Trump winning Alaska given winning New Jersey is lower than given losing New Jersy is because your sample size is rubbish. The chance of Trump winning Alaska given losing New Jersey is an accurate number, the number of Trump winning Alaska given winning New Jersey is like saying "How likely is it Trump wins Alaska given the UK gains US statehood" it's like.... well... if that happens then we're so far outside of what the model thinks can happen then you should be that we're just gonna say it's 50:50 - because who the hell knows.
It's not like saying "Oh well if X swing state goes blue, Y will probably follow", the scenarios in this article are so bizarre that the model should rightly be very cautious and probably default to either refusing to give an answer or just default to 50:50 or the same probably ignoring that data. The implicit bias in this analysis seems to be that if NJ went Red that would be because Trump won by a big margin, but that's not a likely enough scenario to actually get numbers for, and is so unlikely that things like "The supreme court threw out all the ballots for inner city areas" start to become valid possibilities.
A statistical model only has a vague idea of context/the real world. It looks at polls (and probably not really that many polls of Alabama or Mississippi or Alaska) and sees that, statistically, Biden should win 3% of the time or so.
It doesn't have a specific world set of events in mind that would cause that, it just knows that that's how the numbers go, and thus may lead to weird circumstances in the grander results because it has to make the world match the numbers in these small corners.
So, it seems to me that the entire article is predicated on a faulty conjecture, namely that 538 uses a mixture of a normal distribution with an independent heavy-tailed one. (It's not explicitly stated what the author thinks the base model is, but I think "normal" is a reasonable guess.)
I'd be interested in seeing a reverse-engineering analysis of 538's choice of distribution parameters, and extrapolation from there to see if these pathologies still arise with (much) larger samples.
...
That said, ultimately, the choice of how fat to make the tails is a modeling decision, and how the models behave outside the regime of interest isn't as important as how they behave within the operating region. There are key ways we can evaluate goodness of fit once we have results (e.g. bias, MSE) which we can use to determine just how wrong the model was as a predictor, and chances are pretty good that we won't see, say, Trump winning NJ, so we won't actually be able to validate the tail correlation with the vote in PA. But we will be able to validate the correlation in margin between PA and NJ.
Maybe 538's tails are too fat, and every prediction in the 80-95% range ends up going as predicted. Or maybe they're not fat enough, and some races in the 99% bucket end up going the opposite way. Point is, we won't know for sure which models were the best predictors until we can verify the predictions.
(see: all models are wrong, etc. Newtonian mechanics work great as long as your objects are big and slow, for instance.)
I'm not sure why that kind of interstate correlation should impact predictions?
<incoherent rambling :D> IANAS but it feels like these correlations were added to compensate for the failure in 2016 to recognize that state A going one way implied that state B would also go that way. It "feels" like a more correct approach would be to compute some kind of error/weakness measure in a states polls by bringing in those of its geographical neighbors and incorporating the polling error of that entire block vs prior years. Or something.
The intuition I'm having difficulty conveying is that actual voting correlation is based on neighboring states only because you've got bubbles of ideology that aren't strictly cut along state lines. If strength of opinion in a bubble is going one way, then you'll see that mostly in the state at the center of the bubble, but the bubble still spreads into neighboring states, and a "stronger" bubble could push it geographically further into those neighbouring states, and/or could increase the bias in areas inside the bubble. </rambling>
[1] "Always" == most recent history
538 has low positive correlations between states on average, which actually has a big impact, it increases overall uncertainty (and therefore Trump's win probability). Why? If the states are not correlated, you usually end up with a few states going off the rails, like Trump winning Colorado without any nationwide swing.
This implies that correlations increase, rather than decrease, the overall level of uncertainty.
This is also easy to see from a basic probability perspective, using the concept of variance. For example, if you have two coin flips, with outcomes {-1, +1} chosen uniformly at random, then the sum has variance 2 if the flips are independent, but variance 4 if the flips are perfectly dependent.
> the lower the correlation between states, the more uncertainty you need for each individual state forecast to get a desired national uncertainty
Edit: Why the downvote? Each of the between-state correlations can be calculated from 40,000 datapoints.
Additionally, getting worked up about a 3% chance of Biden winning Alabama. I mean, what does a 3% chance even mean for a one off event, compared to a 5% chance or a .3% chance? I know fully well, that it means I should bet $100 if I can get more than $3000 payout, but the trouble is that is only if we bet often enough. (Perhaps often enough on different things.) For a one off thing, the important part is, it is with a very high degree of certainty a loss of $100. So any claims that Bidens chances of winning are too high should be regarded with high suspicion.
Also, I listened eralier to Nate Silver's model talk [0], where he discusses quite a few problems with low quality polls in some states.
[0] https://fivethirtyeight.com/features/politics-podcast-nation...
There are more than enough data points to determine the between-state error correlations, many of which seem to be very off.
> Additionally, getting worked up about a 3% chance
The weird between-state correlations actually have a large effect, they increase state and nationwide uncertainty and as a result Trump has a higher chance of winning.
What value does something like fivethirtyeight add to our democracy, if any? Is this motivation the same as that of diving deep into baseball stats or Star Wars starship engineering, just like “nerding out” for its own sake?
Contrast the voter who never looks at any of these polls with one who keeps up with them daily. Is the latter voter better off in some way? Is this just about trying to read the tea leaves so you can strut and preen later about having been correct, should the dice roll be in your favor?
My concern is that these things are distracting and may actually dissuade some people from voting because they think they “don’t have to.”
Here’s an idea: everyone go vote for whoever you think the best candidate is regardless of what a stack of polls say.
Someone set me straight here, what is the point of all this stuff.
One possible use of polls and election models is for helping people who want to donate to candidates determine which races are closest and where their money is most likely to have an impact.
> Here’s an idea: everyone go vote for whoever you think the best candidate is regardless of what a stack of polls say.
Because the US does not have ranked choice voting in most elections, polls are useful to determine which candidates are viable. If your preferred candidate is only polling at 5%, they are pretty unlikely to win, so you might want to vote instead for whichever of the leading candidates you find most agreeable.
In this system, there are two ways to democratically influence politics by voting:
1. Vote your preferred candidate
2. Withhold your vote from the party more closely aligned with your views, in hopes of helping shift its coalition priorities
If you fall into the second category, accurate forecasting makes a strategic difference.
On the one hand, I think I get your sentiment. On the other, I mean, we are all just solitary individuals floating through this life. Almost all of our important decisions are make at least in part (or more in some cases) dependent on the thoughts and actions of others. That's natural, right? You do otherwise in your life?
If the above is true, it makes total sense why one out of 7 billion plus people would want to understand the choices of others before making theirs.
Great question, and I share some of your concern, though I can imagine some positive framings in addition to what you wrote. For example, to use an analogy, what’s the point of trying to predict the weather, or trying to predict the stock market? There are lots of reasons including planning ahead for likely outcomes, the ability to protect against losses, and last but not least making money.
I can also imagine that the desire to talk about the potential outcomes is valuable as a social activity, and doesn’t necessarily need to meet a standard of influencing the vote, or adding to our democracy.
> My concern is that these things are distracting and may actually dissuade some people from voting because they think they “don’t have to.”
Of course if your concern is founded, this can go both ways... if the polls show the candidate you favor starting to lose, it could be a call to vote.
If polls are distracting and dissuade voters, then unfortunately election results might do exactly the same or worse. When a state has been solidly red or blue and not purple for 50 years in a row, people do (perhaps rightly so) jump to conclusions about the outcome in advance.
One question we could ask is whether, if voting were made mandatory, would election predictions go away? I’d speculate no.
Also, you can make money betting on the outcome itself. If the odds you get are underpriced relative to an accurate forecast, that's a great bet to take.
Furthermore, these forecasts influence where politicians put their focus. Let's say you're Hillary in '16 and you think Wisconsin is yours despite the forecast showing a narrow lead, maybe you should reconsider.
It's an orgy of false precision.
edit: the entire debate is based on the weird assumption that if a prediction about a particular state is wrong, then pollsters must have systematically gotten middle-class Hispanic women over 40 wrong, therefore the odds of other states will change. It's all based in the reification of particular categories that are axiomatically significant for their profession.
> It's an orgy of false precision.
The false precision is pretty obviously coming from you, not the FiveThirtyEight pages that never show more than two (or rarely three) significant figures, and emphasize in every other way they can that the numbers are approximate and uncertain. Have you seen the width of the 80% confidence intervals on their graphs?
As for falsification: all of their predictions are for testable outcomes. We'll always know soon enough who actually wins an election, and which states they won, and by what margin, and who turned out to vote. That's all public record. The only part of the post-hoc analysis that is non-trivial is figuring out how a candidate fared with specific demographic groups. It's imperfect, but between exit polling and precinct-level demographic information and election results, it certainly is possible to detect large pre-election polling errors resulting from inaccurate demographic weighting.
So, wait, you're offended by the election modellers making a prediction, and yet you yourself are making a prediction? What's yours based on? Time machine?
Treating this as a tea-leaf reading (that is, deliberately searching for meaning via free association, without investing it with a truth value) I'm reminded of the "own the libs" meme. I see folks on foxnews.com comments bragging about it; I see lefties complaining about it, but I suspect that it's overblown and not actually a driver behind people's decision-making. But that's what comes up for me when I see "NJ goes Trump" forcing "AK goes Biden".
I'm amused by the resulting thought experiment... if dems started airing "socialists for Trump" campaigns in otherwise safe GOP states, would it move the needle there? Even sillier: if you aired those ads in NJ, would it move the needle in AK?
https://upload.wikimedia.org/wikipedia/commons/thumb/1/1f/Ne...
This is it in 2016:
https://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/Ne...
Long Island is really red there. It’s really hard to say how a democratic stronghold like NYC and something literally a 45 min train ride next to it could vote so differently. Long Islanders are not separate from NYCers, they commute to and work in the city.
To your question, could experiments work in similar situations like this across the country for either side? I think so in the next 50 years as demographics shift (and I don’t think it’s as simple as urban liberals taking over, people do become more conservative as they get older). God knows the dynamic at work between NYC and Long Island in 2016, but it’s obvious things are in flux.
I’ll make a bold prediction here. If Long Island is that red again, yeah, you better believe the typical rust belt states are staying red.
Doesn't that sound like Berkson's Paradox?
Edit: Why am I being downvoted?
It simply wouldn't have been possible for the models to be more accurate with the data they had. As they say, bad data in, bad data out.
That's why this time around the pollsters made sure to be more thorough in their polling.
"This time is different."
I've heard that enough times to be highly skeptical. I'm also deeply skeptical of the notion that polling is even remotely correlated to actual results. Cultural and historical trends play a drastically higher role and are almost always left out.
But it has been strongly correlated to the results in basically all elections so far in all democracies on the planet. Taking 2016 as an example there has been a very strong correlation between polling and the results. The national polling averages were only 3 points off from the actual result. If that's not correlated I don't know what is.
As for fear of doxing: plenty of people openly supported and voted for George Wallace (a noted white Supremacist) back in the day, and even Roy Moore (accused pedophile) just 2 years ago. Proud Boys members openly pose for the cameras even as they espouse racist views, and QAnon members brag about being part of QAnon.
The purported shy Trump voter? Doesn't exist. https://fivethirtyeight.com/features/trump-supporters-arent-...
Not every Trump voter is a "Proud Boy" or noted white supremacist, maybe even some of your friends are who "just aren't interested in politics" because they know that if they were honest with you, you would flip out. Some understand that this attitude about yard signs is actually representative of an entire worldview, and opposing that might actually be the lesser of two evils.
Essentially I think that those narratives about one side "shutting down" the other side are not true for either party. In my opinion they are mainly the result of filter bubbles where you are constantly presented to all the outrageous things that the other side is doing (the above mentioned article is clearly an example) while actually people remain largely civil and are not tied to their political identity in such an extreme way.
It's also likely this works in both directions - if you support Biden, I bet you don't have a yard sign for it if you live in Mississippi.
By construction, the effect is strongest the more you're not in the majority, which also means your unspoken support is more likely to not matter on the actual outcomes of the election.
Newsflash, they aren't, so your hypothesis must be they're too shy to admit their Trump support to a machine?
Everybody is just fighting the last war where Trump suddenly won for a couple of reasons. So, the Dems are scared they are missing something, and the Reps are going "Haha, who cares about modelling".
Will, the forecasts be perfect? Nope. But is the margin rather large, but not unsurmountable? Yes it is. Are the mistakes from last time repaired? Yes, they take care of uneducated whites. Is there evidence Trump has found a new source of voter support? I haven't seen it.
If anything, Trump supporters have been extremely vocal about who they were supporting, to the extent that they frequently violate social norms and try to takeover events and gatherings to make their political affiliations known, like this week with Among Us.
There is a 0.750.750.75*0.75 or a 31% chance of no rain at all
Personally I would have put it about 60-40 Hillary-Trump.
From my point of view, this is only because said people considered Trump winning so extremely unlikely that 538 getting it sort of right appears exceptional. In reality, they were still quite wrong, just slightly less so. Ergo I don't see much value in their model.
I'm seeing the same exact thing today with Biden at a 90% chance of winning.
https://projects.fivethirtyeight.com/2016-election-forecast/
If I had a laptop that only worked 1/4th of the time, rather than 1/20th of the time, would that make it a reliable laptop? I don't think so.
Also, it doesnt make sense to look at a single prediction to evaluate a model.
Out of all the predictions they have made (did you look at individual state predictions?), how many were correct (and how confident were they?) - how many were wrong (and how close to 50% were they?).
That is how you evaluate a model (aka cross entropy)
For example: anyone paying attention to the Rust Belt ±1980-2016 would have dramatically upped Trump's chances in Pennsylvania and Michigan. FiveThirtyEight had Hillary with 70%+ chance of winning both, which to me, shows a deep ignorance of actual cultural factors.
There was a very decent chance that Clinton could have won in 2016 (if any factor had gone slightly better for her), and if that had happened, nobody would be saying this now. This is literal hindsight bias.
My view is simple: the media completely, totally got 2016 wrong, mostly for sociological reasons. The people making the predictions simply had a huge blind spot. Brexit is another similar situation. The fact that Hillary almost won or Brexit almost didn't happen isn't really the point, because both things were never expected to be even remotely that close. Had the predictions been "Pennsylvania will be close", it would be relevant, but those weren't the predictions.
How else would one predict the outcome of a die roll, specifically?
It's unfortunate we can't just run the election again a few times, and actually find the rate at which Trump is elected given the polls.
And it's not empty signalling if 538 assigned Trump a higher chance of winning; they were pretty much the only ones saying he has a chance. That is why people think the models are useful.
Likewise asserting that California with a 3% chance of going Trump is absurd is an unreasonable degree of overconfidence. Assuming maximizing expected return, the author is implying that they would be willing to take a bet that Trump would lose California with odds >> 97::3, i.e. presumably they would take a bet where I bet $1 to every $99 they bet. To be critical of a model based on outcomes it predicts with tiny probability you need truly remarkably biased priors.
I would be more than happy to make this bet with anyone willing to take the other side - as in literally, find a modern middle-man system and I'm game.
You may want to be aware that you have provided me a nontrivial arbitrage opportunity, as the odds on predictit are closer to 7:93 : https://www.predictit.org/markets/detail/6611
You’d have to be crazy to take this one specific bet, you can only realistically take all possible improbable bets.
However, for my use of 538, I’m perfectly happy to ignore such scenarios (such as Trump taking New Jersey). I can call the election in his favour by myself in these scenarios without needing the model.
Why is there so much fascination with polls to begin with? I understand that there are betting markets, but it seems sort of silly. If you had a 100% accurate poll, for instance, then what would be the purpose of the actual election?
Polling is useful for candidates and gives them ideas on where to target outreach and spending.
For the rest of us, it gives us something to watch. With Presidential campaigns running for almost two years in advance of the actual voting weeks, there’s a huge gap between when the thing starts and when we see results. This way, people have something to fill the time. Even now that voting has started, we are still another 10 days until the voting is done and likely another 7 after that until we have a sufficient count to know who has been elected.
That’s a long time for a populace that’s worried, distracted, and interested, especially since so many of us live in states where—due to the mechanics of a broken election system—we can’t do much to influence the national outcome.
The only way that would work is if he made people more likely to lie about their voting intentions. Now, there may be something about Trump that makes polling methodologies less accurate (notably, many pollsters have started to take into account education, which turned out to be unexpectedly important last time round) but that points just to bad methodology, not inherent unpredictability.
To run a 100% accurate poll would require you to sample every voter, so it would literally be an election.
Actually an election is literally a poll in the sense of a sample, you are counting people at "polling stations" in order to gauge the public mood about who should be president.
When you see it that way, Nate Silver is predicting a sample of an unknown distribution, the "true" distribution of people's preferences.
The fact that you can only make one officially binding sample ("the voters") is a practicality, as is the fact that there's an electoral college that means votes have different values. The fact that turnout matters is another issue, a statistician might call it a sampling problem.
And apparently columbia.edu does not fulfill those criteria.
538 gave Hillary a 71.4% chance of winning
https://projects.fivethirtyeight.com/2016-election-forecast/
I think that's what he's talking about.
He's basically saying he doesn't believe the die is fair/unweighted.
So stating the odds of a fair die is kind of immaterial to his point. We need to demonstrate to the guy that the die is fair.
Someone else posted a link (I think) of 538 going over on how accurate they were. Whether their odds bore out. Here it is: https://projects.fivethirtyeight.com/checking-our-work/
Basically what they did was bucket every prediction by odds. If they predicted 70/30, it went in that bucket. And they're "right" at about the rate of their predictions. In other words, for every 70/30 prediction they made, the people/teams with 70% chance to win, won about 70% of the time.
That shows that 538 in this case is pretty decent at calculating odds.
Grandposter doesn't believe the die is fair. That's a different argument than the guy I responded to made.
Incidentally, this trick is something magicians sometimes do. Sometimes when a trick has gone wrong they'll make a wild guess. If they're right, the audience is impressed. If they're wrong, they'll brush it aside with some joke and the audience won't notice/mind much. This works for things card guessing tricks and puedo-psychic/cold reading stuff.