So what can we learn from Nate Silver's mistakes?

News

  • Author: Carlos Alberto Gómez Grajales
  • Date: 14 May 2015
  • Copyright: Image appears courtesy of Getty Images

You may have been closely following the UK's General Election. It became particularly interesting for statisticians back in March, when Nate Silver announced that his team would be forecasting the election results [1]. Silver became famous with his 2008 predictions in the US, where he accurately predicted Barack Obama's victory. He did it hours before exit polls could, by using a statistical model he devised in March of the same year [2]. He's been forecasting American elections since then, with mostly positive results.

thumbnail image: So what can we learn from Nate Silver's mistakes?

Yet this time, he was entering, literally, into foreign territory. Nate Silver was now forecasting an election with a different political and party system. How did he do? Well, you already know the answer, since these may very well be one of his most publicized results. His model failed to predict the number of seats each party would win, by a wide margin. The day before the election, FiveThirtyEight forecast that the Conservatives would get 278 seats, Labour would get 267, the SNP 53 and the Liberal Democrats 27. It was way off. The actual final results were 330 seats for the Conservatives, 232 for Labour, 56 for the SNP and just 8 for the Liberal Democrats [3].

Many explanations have been written regarding why these forecasts failed. But before discussing Silver's explanations, along with some of my hypothesis, it is worth to take a look at FiveThirtyEight’s model, since it is a clever exercise of statistical analysis. It shouldn't be so easily dismissed, as his methods have been proven useful in some other latitudes.

Here's how FiveThirtyEight's UK model works. First, they gather information regarding the polls in the UK. Most of those are national polls, since they tend to be the most common, usually commissioned by the media. By averaging those polls, usually weighting them based on sample size, previous pollster accuracy, etc. you get one first prediction. The second step involves analyzing the correlation between polling and actual electoral outcomes, based on previous elections. That way the polls are now weighted based on their estimated prediction ability [4]. After that, we get an estimated national voting share for each party, based on all the work we did with the national polls.

The next step involves translating that into seats, meaning that we need to extrapolate those national numbers to constituency level predictions. Luckily, for these elections, the analysts had enough constituency level polls to work with, thanks to the generous contribution of Michael Ashcroft, a Conservative in the House of Lords who had been personally commissioning constituency polls for over a year [4]. By working with these surveys, just as they did with the national polls (though with much less historical data), they could obtain an estimated voting share for each party at the level of constituencies. You can then use that information to predict the number of seats each party would win. The analysis team didn't have enough information for every single constituency, so they used a mixed regression model [5] to complete their gaps. As a final step, they adjusted constituency level estimations, to make sure the voting share matched the one estimated through the national polls. After that, they had a prediction for elections in all constituencies, which yielded their final forecast.

Notice that this model is quite similar to the one FiveThirtyEight used in previous elections, a model that has produced accurate forecasts before [6]. The main difference would be the amount of information available, which would result in an increased number of corrections/adjustments. Also, it is worth remembering that Nate Silver is not alone in the business of predicting elections. Some other scientists have devised their very own models and statistical schemes to predict future elections, mostly with good outcomes [7].

So, what went wrong? Why was Silver's model so off in the UK? Well, first of all, most of Nate Silver's work has been devoted to American politics, yet UK elections are a totally different beast. From the number of parties to the political system itself, many of the lessons learned in the US could not (or should not) be applied in British territory. I assume many of the usual adjustments Silver made to polling data or historical results, couldn't be as thorough this time.

The modeling team also discussed some possible reasons for the mistakes. Their model predicted quite an inaccurate national vote share. Now, if you remember, the national voting numbers were estimated based on results from the national polls, yet those were not quite reliable as it seems. The polling average had Labour and the Conservatives with an even voting share on the night before the election. This was not just the average of the polls, it was the consensus. Nearly every pollster’s final poll placed the two parties within 1% point of each other. On the final results, the margin was above 6% [3]. The reason why the polls were so off is another topic entirely, worth discussing in itself. In the meantime, let's just say that polling is quite a difficult endeavor - I've done it quite a few times myself to know it. Electoral polling involves not only technical knowledge on statistics, politics and complex survey analysis, it also demands you to have sufficient information that is current and reliable enough in order to build a good sampling frame, to ensure an appropriate sample selection and to aid fieldwork operations. Additionally, a poll estimate should consider factors such as weighting, non-response adjustments, under coverage and even voter turnout - all this whilst working within an extremely tight schedule and budget. Most of the times a poll does not give you the best available estimate, but the best affordable one.

Besides all the caveats and details you have to consider to make a good poll, there's another detail that is usually overlooked, even when it can dramatically change estimations. That is wording. Believe it or not, how you phrase a question in a poll can drastically affect the results you get [8]. There has been much research in the area, though it is more didactic to learn from FiveThirtyEight's case. As previously explained, the model gathered information from constituency-level polls. These particular polls included two variations of the usual voting intention question. The first one, used the typical and commonly used wording: “If there was a general election tomorrow, which party would you vote for?” A second variation included a more complicated one: “Thinking specifically about your own parliamentary constituency at the next general election and the candidates who are likely to stand for election to Westminster there, which party’s candidate do you think you will vote for in your own constituency?” For reasons that don't go beyond “indirect evidence”, Nate Silver's team decided to use the second variation in their analysis. That second wording is not only longer, but also confusing: it uses longer words and it is harder to read. Research on questionnaire design would oppose to such wording [8]. According to FiveThirtyEight's blog post, just by using the first, simpler wording, their model would have produced a more accurate prediction - still not great, but noticeably better than their published forecast [3]. To be fair, Nate Silver and his colleagues later defended their choice, claiming that, under some metrics, the second version of the question was indeed better, even if it yielded a worse prediction [9]. I'd be skeptical about this, but the fact remains that the simple words used in the questionnaire had such an impact in the forecast. This should be a remainder for all researchers: wording is crucial in social surveys.

Many could dismiss forecasting efforts as a waste of time, particularly after such a disastrous outcome. But please, do not forget that scientific literature in the political landscape is still shaping up and these exercises are an invaluable collection of knowledge. In the end, Nate Silver's forecasts are a scientific enterprise. Even if the predictions are awfully incorrect, it shed a light in some of the methodological flaws polls and political studies have. We have learned from his mistakes that wording is important in a questionnaire and that more care should be put in the design of voting polls. Even if FiveThirtyEight could not accurately forecast each of the seats, we should applaud them for trying, failing and allowing us to learn with them along the way.

[1] Silver, Nate et al. We are forecasting the U.K. General Election. - FiveThirtyEight Website (March, 2015)
http://fivethirtyeight.com/datalab/uk-election-forecast/

[2] Clifford, Stephanie. Finding Fame With a Prescient Call for Obama – The New York Times Website (November 9, 2008) http://www.nytimes.com/2008/11/10/business/media/10silver.html?pagewanted=all&_r=0

[3] Lauderdale, Ben. What We Got Wrong In Our 2015 U.K. General Election Model. - FiveThirtyEight Website (May, 2015) http://fivethirtyeight.com/datalab/what-we-got-wrong-in-our-2015-uk-general-election-model/

[4] Lauderdale, Ben. How Our U.K. Election Forecasting Model Works. – FiveThirtyEight Website (March, 2015)
http://fivethirtyeight.com/features/how-our-uk-election-forecasting-model-works/ 

[5] Multilevel model - Wikipedia The Free Encyclopedia
http://en.wikipedia.org/wiki/Multilevel_model

[6] Silver, Nate. FiveThirtyEight’s Senate Forecast. – FiveThirtyEight's Politics (US Nov, 2014)
http://fivethirtyeight.com/interactives/senate-forecast/

[7] Grajales, Carlos. Media look to statisticians to increase readership. - StatisticsViews Website (November, 2014)
http://www.statisticsviews.com/details/news/6931591/Media-look-to-statisticians-to-increase-readership.html

[8] Grajales, Carlos. Singing surveys and Jedi knights: How not to ask questions. – Significance Magazine (Aug 2013) DOI: 10.1111/j.1740-9713.2013.00682.x
http://onlinelibrary.wiley.com/doi/10.1111/j.1740-9713.2013.00682.x/abstract

[9] Lauderdale, Ben. What Else We Got Wrong In Our 2015 U.K. General Election Model. - FiveThirtyEight Website (May, 2015)
http://fivethirtyeight.com/datalab/what-else-we-got-wrong-in-our-u-k-election-model/

Related Topics

Related Publications

Related Content

Site Footer

Address:

This website is provided by John Wiley & Sons Limited, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ (Company No: 00641132, VAT No: 376766987)

Published features on StatisticsViews.com are checked for statistical accuracy by a panel from the European Network for Business and Industrial Statistics (ENBIS)   to whom Wiley and StatisticsViews.com express their gratitude. This panel are: Ron Kenett, David Steinberg, Shirley Coleman, Irena Ograjenšek, Fabrizio Ruggeri, Rainer Göb, Philippe Castagliola, Xavier Tort-Martorell, Bart De Ketelaere, Antonio Pievatolo, Martina Vandebroek, Lance Mitchell, Gilbert Saporta, Helmut Waldl and Stelios Psarakis.