Comment by ninth_ant
9 hours ago
> I did too much bragging in the media and didn’t anticipate the extent to which public opinion toward FiveThirtyEight would shift once we became a corporate-backed incumbent rather than an eccentric upstart
Can’t speak for everyone else, but it wasn’t this for me. It was about 2016 presidential that lost me.
He tries to justify this later about how theirs was better than other outlets but I don’t care. Call it emotional, naive, unfair or whatever you want, but regardless I had zero interest in reading any of their predictions or analyses after that.
Not even mad, just that to my experience they had one job and they didn’t fulfill it at the most important time. They went from appearing insightful to just one opinion amongst so many others.
I had complaints about 538, especially the early days, but don't understand this critique at all. A 30% chance hitting is completely unremarkable, and it was a perfectly reasonable reading of the evidence at the time. Nate isn't wrong that conventional wisdom was way off, with even supposedly statistical models giving Hillary a 99% chance of winning.
Elections, like many things, have some inherent uncertainty. A several point polling error is normal, so a candidate who is down a couple points on election day has a decent shot of winning.
Discussion of stats models is always complicated by the fact that a lot of people will read "30%" as a "no" prediction and claim your model is wrong if the thing happens. On the one hand, one strategy is to "hide" the numbers a bit behind a blaring headline that says "we are not sure!!" It's a bit of an art to decide when to be "sure" or not. On the other hand, in research for example you can just say screw it, I care if the correct people are correct, not if a bunch of wrong people are wrong.
I feel like the correct strategy for 538 when it was actually niche was to be precise, but then it went viral and maybe should've hit the IDK button much harder and more often after that.
The real caveat is that 538 was a Monte Carlo model, and is only as good as its inputs. "Here's what the current spread in polling numbers is *given our model and the current polling and their reported uncertainties.*" Polling uncertainties are themselves computed under certain models, and those models are subject to errors. I don't think 538 hid this, but it's a difficult caveat for people to reason about because the sorts of modeling errors that have the most influence usually represent "unknown unknowns".
3 replies →
That's a core mechanic in games like Dispatch.
People don't like seeing a 95% chance of winning and then losing. The game tweaks the odds, so certain thresholds become gimmes (something like "if the displayed odds are better than 75%, treat them as 100%").
4 replies →
> Discussion of stats models is always complicated by the fact that a lot of people will read "30%" as a "no" prediction and claim your model is wrong if the thing happens.
I've even heard things like "70% chance of Hillary winning means she gets 70% of the votes!" (and tangentially, my far-too-long argument with someone on Reddit who insisted "there is no way in hell 50% of the people in this town make above the median income"...)
I don't understand why this is surprising. People didn't go to FiveThirtyEight to marvel the science behind it. The science was just supposed to give you what you came there for: the actual election results.
In the end, it turned out that predicting elections is still very hard, and that for all the fanfare, FiveThirtyEight performed only slightly better than what you could find in any other reputable newspaper, so it kinda lost its appeal.
> FiveThirtyEight performed only slightly better than what you could find in any other reputable newspaper
FiveThirtyEight gave Trump double the odds of the next highest reputable prediction, which was The New York Times Upshot (15%). Princeton Election Consortium gave Trump less than 1%.
That is not "only slightly better" to anyone who's statistically literate.
A FiveThirtyEight reader in 2016 was significantly better calibrated regarding Clinton’s chances than a reader of other reputable newspapers.
4 replies →
> it turned out that predicting elections is still very hard
So maybe we shouldn't be doing it. The value of predicting an election in the large out in public seems kind of dubious, and it's more like gambling than actually being useful. A candidate only runs, and continues running, if they think they can win. All predictions like these do is confuse voters leading up to election day and while they are voting. It keep candidates from making strong cases for their platform, keeps the voters from listening to the candidates' platforms, and encourages team-based partisan politics.
1 reply →
538 was never about magically making polls more reliable, and only people that don't understand what polls are could think that (caveat: lots of people don't understand how polls work).
538 was about analyzing and communicating the information from those polls in an easily accessible form. If you came to the site for that, you weren't mad that they "predicted poorly something that was impossible to predict from the data sources they used" ... you were just mad at Trump for winning (despite polls suggesting otherwise).
6 replies →
Polls became much less interesting as an Entertainment category once we all had experience with how unreliable they are.
I can't find the source anymore since 538 is no more, and I recall Nate even describing what could (and did) happen, which was that one swing state moving to the right had a high likelihood of them all moving to the right.
Yeah, Nate has talked a number of times about polling errors being correlated across states. In fact that's probably one of the most common mistakes models can make, treating correlated inputs as independent. There's a long history of that mistake in financial markets as well.
In 2024 the single most likely outcome his model had was trump winning all 7 swing states. The second most likely was Harris winning all 7.
Not just that, predictions also impact voter participation.
I think this is all true, but it dodges the bigger issue. A presidential election has a binary outcome: yes/no, win/lose. If your statistical model doesn’t contain this single bit in its output, then it doesn’t meet the minimum requirement for being a prediction.
Now you might say that it was on me as a consumer to understand this in 2016, but I remember the look of total shock on Nate Silver’s face when he called the winner on live TV that night, so clearly he didn’t really understand it either. Lesson learned for all of us, I guess.
It was 30% in the end, before the nomination it famously gave him a 2% chance of getting nominated. All the talk about 30% is disingenuous
https://en.wikipedia.org/wiki/FiveThirtyEight#2016_U.S._elec...
They had him at 30% at a time when most reputable media had him in single digits at best. You might not remember now, but the result of that election was a real shock.
If I say you have a 50% chance to win a coin flip and you lose it, that doesn't mean I'm wrong.
A key thing though is 538 did regularly test the calibration their models: https://web.archive.org/web/20190410030104/https://fivethirt...
> "What you’ll find, though, is that our calibration has generally been very, very good. For instance, out of the 5,589 events (between sports and politics combined) that we said had a 70 chance of happening (rounded to the nearest 5 percent), they in fact occurred 71 percent of the time. Or of the 55,853 events that we said had about a 5 percent chance of occurring, they happened 4 percent of the time."
Huh. I first started listening to 538 in the run-up to the 2016 election and started really paying attention to them precisely because their 30% figure was so much higher than all the other polls. It was shocking to me then (and still is now reading your comment) that people didn’t seem to understand that 30% in the context of that particular election and that particular candidate suggested a remarkably high chance of winning, not a really low chance of winning. It’s a strange thing where people seem to think that less than 50% = not happening.
I really think a majority of NYTimes and ABCnews consumers don't know the difference between a 2/3 chance (super close) of winning and 2/3 of the vote (a landslide).
Apparently neither do a big chunk of HN readers.
That’s like saying “there was only a 30 percent chance of rain today and it rained, so I will never look at the weather forecast again.”
Because they said trump only had a 30% chance to win?
What if they had said 49%? Would that have made their prediction worthless?
I mean everyone said he had a snowball’s chance in hell and then we ended up with him for two terms because the Democrats can’t stop fighting over the worse possible candidates to back that no one is asking for.
Everyone said that except for 538. That's why 538 was worth reading.
I find it interesting to blame it on the democrats. We ended up with him because enough people voted for him.
6 replies →
do you think nate silver is part of the problem or part of the solution?
the turnout-of-demographic-groups-based election model is surely the underlying intelligence failure here.
Democrats will not let people choose candidates because that may be too dangerous for their interests. We'll never get good candidates as long as the current leadership is in control.
4 replies →
When pressed before the election, Silver did not explain where Trump's much higher probability of winning came from. He predicted a Trump loss, Trump won, and he claimed victory because he gave Trump a better chance of winning. There's no way that strategy could have failed.
Silver claimed that his model was better because it predicted a high correlation between PA/MI/WI.
A model that predicts a 30% chance of winning the election will be wrong 1 out of 3 times, which is not quite a coin flip but close enough.
Nate Silver is not a magician! He can't magically make polls reliable!
All he (or anyone) can do is interpret or analyse poll results, and then surface their findings in a way a larger audience can understand. 538 did that better than any other poll analyst ... but they all got it wrong because the polls themselves were faulty.
TLDR; You can't get water from a stone, and no one (not even Nate Silver) can get perfectly accurate predictions from (inherently flawed) polls!
2 replies →
We should have a drinking game in Nate Silver thread anyone complains about 2016 prediction. Then everyone piles on to point out how probabilities work.
I long ago stopped commenting on threads (in /r/CFB) about the SP+ college football model because it go so, so, so tiring correcting the exact same misunderstandings about things as fundamental as what the model was even purporting to do let alone, how it worked. It is incredibly disheartening to me to see the staggering level of statistical illiteracy on display in any online conversation about a statistical model. I've come to the conclusion that it's not worth my, or any one else's time. I'm glad these things exist, and the communities who want them will self select. It's not use trying to convert haters.
It wasn't even a complaint, just a personal anecdote to help share some context as to why the site may have failed to retain consumer interest post-2016.
But yes I'll join you with the liver damage and drink 17 shots.
538 used the example of Trump having approximately the same chance of winning the 2016 presidential election as the Cavaliers had of winning the NBA championship round vs the Warriors. Both Trump and the Cavaliers won with a ~25% predicted chance.
538 made very clear with this analogy that both Trump and the Cavs were underdogs, and that both had a solid chance of winning.
A dice roll has a 16.6% chance of landing on any given side, meaning an 83% chance of not landing on that side.
If you guessed a "two", and it landed on "two, I wouldn't really be that impressed, even though there was an 83% probability going against you.
I think this gets to the core of why a lot of this election prediction stuff doesn't work. People just don't parse the numbers the way the authors intend.
FiveThirtyEight had Trump at a 30% chance of winning, and he won. The model wasn't wrong. The less likely of two outcomes occurred. Even if they'd had him at 1% they still wouldn't technically have been wrong though I think complaints might be more warranted.
If they had Trump at 49% would you have still been angry? What about at 51%? Would it have been okay then?
Technically this is right. But if that is the case (and it seems to be), then a coin flip is better than their models. Because we only care about the current election, not a sequence of 1000 elections (which will not happen, by the way).
So use a coin flip to predict the weather then
1 reply →
> But if that is the case (and it seems to be), then a coin flip is better than their models.
If a coin flip is the necessary mental model to remind you both things can happen, then sure.
People just love horse race coverage. Silver gave us the most accurate horse race coverage. Maybe the lesson is stop following horse race coverage.
But most people went back to the tea leave readers. That way when the election was over, it can justifiably be the charlatan's fault that viewers got over-invested in their predictive capabilities.
I don’t disagree, I think the tea leaf reading is ultimately pretty futile.
But at the same time I do think it’s valid to say it’s more than a coin flip. The polling data over the election cycle showed that Trump had a smaller but still legitimate chance of winning. The data was different in 2020, when he lost.
FiveThirtyEight gave Trump a 30% chance. Their reporting did make clear that with margin was within the range of a normal polling error. And sometimes you get more than a normal polling error.
It doesn't help that the US has a terrible election system that often leads to small margins in some states being decisive.
> FiveThirtyEight gave Trump a 30% chance.
I know I'm being super conspiratorial here but why wouldn't all forecasters predict just between 30% - 70%? That way if they're "right" they can take the credit for it and if they're wrong they can say "well, we weren't that wrong". That's probably what I'd do anyway...
It implies a close race or a strong reason to believe there's some sort of systemic polling miss, and if it's a blowout you still look pretty bad. Especially if you don't have some kind of good explanation for the miss/you keep making those kinds of misses frequently.
Also there's more going in those forecasts besides just the "% chance to win". There's expected results in terms of %'s of the vote for the candidates, and that's what people tend to focus on for actually analyzing your performance and credibility after the fact.
You getting the outcome correct but being off by 20 points on the margin is a much worse performance than you getting the outcome wrong but being within 0.5 points of the margin. (ex: Results are 49.75/50.25, you predicted 30/70, another outlet predicted 50.25/49.75).
538 claimed that they post-checked thousands of elections and their percentages were pretty close. (E.g., 30% chances happened about 30% of the time, million to one chances happened every single time)
Of course there's more than that. Predicting higher uncertainty than warranted would be a different failure in the model. But that didn't really happen in that election.
> I know I'm being super conspiratorial here but why wouldn't all forecasters predict just between 30% - 70%? That way if they're "right" they can take the credit for it and if they're wrong they can say "well, we weren't that wrong". That's probably what I'd do anyway...
For anyone making many predictions, you can analyze the outcomes to see how accurate those percentages are.
For anyone making few predictions, you should never trust their track record even if it's technically perfect.
You're completely right, although I believe this number was not decided personally. They just happened to pick algorithms that have this "nice" property because it will lead to the same result.
> Call it emotional, naive, unfair or whatever you want
Yep definitely all those.
Why is it so hard to admit 30% is not 0%?
As a younger man I would have been with the commenters mansplaining probability, but I've aged into realizing that thinking of the election like a marble pulled from an urn whose contents we have probed with polling is just as bad as thinking of it as deterministic. The reason people read fivethirtyeight, probability-savvy or not, was almost entirely to be told what was going to happen, which is sort of incompatible with feeling you can do anything about it. In that way it's probably worse than old-fashioned pundit-driven horse race coverage because it has an air of scientific authority.
> Call it emotional, naive, unfair or whatever you want
We would need a pass from the mods lol.
If you use 538 data, on average you make money betting. Its more correct than not
Nate was a huge outlier in that prediction, he gave trump a better chance than almost anyone else that I can recall, so why are you mad at him about that?
What made me mad is Nate seemed to turn into a MAGA troll himself after that election.