Since it began polling in 1992, SurveyUSA has forecast the outcome of 780 separate election contests (as of 09/01/07). Many of the nation’s most respected commercial and academic pollsters also polled on these same election contests.
One way to evaluate how good a pollster is, is to see how close a pollster’s final pre-election forecast comes to the actual outcome of an election. If a pollster’s final estimate is consistently at odds with the actual outcome of the contest, that tells you one thing about the pollster. If the pollster’s final forecast consistently comes close to the actual outcome of an election, that tells you something else.
If more than one pollster polls on an election contest, which is often the case, then it is possible to compare each pollster’s final pre-election forecast to the outcome, and determine which pollster had the most accurate forecast. That is what SurveyUSA has done here.
Every time SurveyUSA works on an election, all competing pollsters are identified. SurveyUSA’s final pre-election poll is compared to all competing final pre-election polls (if competition’s final poll was released during final month of campaign; earlier “stale” polls are ignored). SurveyUSA’s outcome is then judged to be more precise, less precise, or comparable to the competing pollster’s final poll.
Sounds like a simple exercise, yes? Well, yes and no. Not all pollsters agree on how to measure the precision of a pre-election poll. Some prefer to measure precision in one way, others prefer to measure precision in a different way.
SurveyUSA did not want to create possible controversy (or apparent bias) by choosing just one way to measure its performance. SurveyUSA was concerned that others might say SurveyUSA had chosen a method of measurement that showcased SurveyUSA results in a favorable light, while deliberately avoiding a different measure that might have showcased SurveyUSA’s results in a less-favorable light.
As a result, SurveyUSA created this Interactive Election Scorecard, which includes all possible ways to measure the precision of competing pollsters. The Scorecard lets you look at SurveyUSA’s performance using the measure (or measures) of your choice.
SurveyUSA does better using some measures, worse using other measures. The good and the bad is all here, for your inspection and review.
The historical data contained in this database was gathered from available archived news reports at http://www.lexisnexis.com/ , from http://www.pollingreport.com/ and from the fee-based service Hotline, http://www.nationaljournal.com/ . More current data was gathered from the Internet websites http://www.pollster.com/ and http://www.realclearpolitics.com/, both of which attempt to present a compendium of released opinion polls.
- No SurveyUSA election polls are omitted from this analysis.
- No competing polls or pollsters are intentionally omitted.
- Every attempt is made to include all known polls, and to ensure the accuracy of information about competing polls.
Requests for supporting documentation, or notice of omission or inaccuracy, should be made to firstname.lastname@example.org .
The High-Level Overview
What is the Significance of this Chart?
When SurveyUSA conducts election poll research, it uses TV news anchors to secure respondent cooperation. No other research firm interviews poll respondents this way. Other firms use headset operators.
Because SurveyUSA’s approach is innovative, some want you to believe it is inferior. This interactive tool provides data to test that hypothesis.
This Interactive Election Scorecard is not a marketing document.
It does not selectively include only SurveyUSA’s best work, nor does it selectively exclude the competition’s best work. Rather, the Scorecard is exhaustive: every SurveyUSA election poll is included, good and bad, since SurveyUSA began polling in 1992, and every known poll conducted by a competing pollster is included.
In this way, the Interactive Election Scorecard is a scientific measurement of SurveyUSA’s performance.
I invite you to please stop now, and search the Internet. You will see that no other pollster, academic or commercial, publishes a Scorecard similar to this.
What does “SurveyUSA Winning Percentage” Mean?
SurveyUSA calculates “winning percentage” the same way the National Football League does. One point is awarded for a “win.” One-half a point is awarded for a “tie.” No points are awarded for a “loss.” The number of points are divided by the number of contests.
- If SurveyUSA had competed against Pollster “A” 10 times, and SurveyUSA was better 10 times, Comparable zero times and Worse zero times, SurveyUSA’s winning percentage would be 1.000. (10 points divided by 10 contests).
- If SurveyUSA had competed against Pollster “B” 10 times, and SurveyUSA was better 5 times, Comparable zero times, and Worse 5 times, SurveyUSA’s winning percentage would be 0.500. (5 divided by 10).
- If SurveyUSA had competed against Pollster “C” 10 times, and SurveyUSA was Better zero times, Comparable zero times, and Worse 10 times, SurveyUSA’s winning percentage would be 0.000. (Zero divided by 10).
Obviously, the higher the Winning Percentage, the better. A score of 0.500 is Average. Any score higher than 0.500 is Above Average. Any score lower than 0.500 is Below Average.
Why is there more than one measure of election-poll accuracy?
If you want to compare one meteorologist against another, life is easy. Weather Service A says the high temperature tomorrow will be 53 degrees; Weather Service B says the high will be 57 degrees. The actual high turns out to be 57 degrees. B is more accurate. Case closed.
Polling is more complicated. Consider a hypothetical election between two candidates: Smith and Jones.
Pollster A says the outcome is going to be Smith 57%, Jones 38%. Pollster B says the outcome is going to be Smith 48%, Jones 32%. Election Day comes, and the actual vote shows that Smith wins 57% to 43%. Which pollster did better?
Pollster A says, “I did better. I said Smith would get 57% of the vote and he got 57% of the vote; I was perfect.”
Pollster B says, “Wait a minute: Smith won by 14 points. I said Smith would win by 16 points. You said Smith would win by 19 points. I was off by 2 points. You were off by 5 points. My poll is better.”
A dispassionate observer named “C” comes along and says, “Not so fast. Pollster A did not account for 5% of voters, who Pollster A called ‘undecided.’ And Pollster B did not account for 20% of voters, who Pollster B called ‘undecided.’ If you take out these undecided voters, then normalize Pollster A’s forecast to 100, and normalize Pollster B’s forecast to 100, what you find is that pollster A actually said the outcome of the contest would be 60% to 40%. And Pollster B also forecast the outcome as 60% to 40%. So, in fact, it’s a tie.”
Another dispassionate observer named “D” comes along and says, “Just a minute there ‘C’. Who are you to say how the undecided voters in Pollster A’s poll are to be allocated? And who are you to say how the undecided voters in Pollster B’s poll are to be allocated? I see a different way of solving the problem…”
And so on, and so on …
For this reason, there is not just one way to measure which pollster is best. And for this reason, SurveyUSA here includes all of the different ways of measuring the accuracy of an election poll, so that no one can accuse SurveyUSA of only including those measures favorable to SurveyUSA.
Do you breakdown how SurveyUSA has done against specific polling firms?
Yes, SurveyUSA alone shows you how SurveyUSA has done on those occasions when it has competed head-to-head against the country’s most respected polling firms. This “IES” (Interactive Election Scorecard) tool is designed to be used interactively, with you in complete control of the display.
But just to give you an idea of what the data that is available to you looks like, here is a still-image of the tool when it is opened:
Who is Mosteller?
Following the 1948 presidential election, a commission was formed to study the failure of polls to predict Truman’s reelection. The resulting book, The Pre-election Polls of 1948: Report to the Committee on Analysis of Pre-Election Polls and Forecasts, by Frederick Mosteller et al, was published by the Social Science Research Council (New York, 1949). In this book, eight different ways to measure the accuracy of a pre-election poll were proposed (for short-hand herein and going forward: “Mosteller 1″ through “Mosteller 8″). The last two of these measures, “Mosteller 7″ and “Mosteller 8,” are not included in this Interactive Election Scorecard, because they depend on information that is either irrelevant or not compiled (electoral college vote prediction, for example).
Though it has been more than 50 years since Mosteller proposed his measures, they remain today the “default” method by which pre-election polls are evaluated. The measures “Mosteller 1″ through “Mosteller 6″ are defined on page 55 of the 1949 book; they have been adapted here to be applicable to all elections (not just those in which one Democrat runs against one Republican).
SurveyUSA also includes a measure, presented by Traugott et al at the 2003 AAPOR Annual Conference, and a measure developed by Joseph Shipman , PhD, presented at the 2005 AAPOR Annual Conference.