Nick Panagakis, whose firm, Market Shares Corporation, conducts polls for the Chicago Tribune and WGN-TV, takes issue, at Pollster.com, with the way SurveyUSA measures error in our Pollster Report Cards:

My colleague Jay Leve does himself and others an enormous disservice by using a method which says if Pollster A says Smith beats Jones by 8 points, and Smith in fact beats Jones by 4 points, then Pollster A has an error of 4 (8 minus 4). A 4 would be recorded in the table at their site for that contest. The Survey USA site says there are limitations to this and other measures of pollster accuracy. You bet there are.

This is because it is the poll estimate that is subject to sample error, not the margins; e.g., 48% voting for A and 52% for B would be a 4 point margin. If the election turns out to be 46%/54%, an 8-point margin, the margin error is 4 points. The error of the estimates in this case is 2 points, half the margin error. Differences between poll and election margins in statistical analysis, or error of the margin, should not be used.

In summary:

1. Elections are zero-sum games. This means that two points high for one candidate MEANS two points low for the other. So estimate error is the more valid measure. Estimate errors are not additive which is the effect of using the difference between election and poll margins.

2. This is also the only error measure that can be compared with sample margin of error always included in poll reports. Whatever method is used should be comparable to stated statistical margin error. Only the error of the estimates does that.

Posted by: Nick Panagakis | January 27, 2008 9:49 AM

Context?

Mr. Panagakis, active at the National Council on Public Polls (NCPP), is concerned that SurveyUSA accounting might be mis-interpreted in a way that reflects poorly on all research professionals. To the extent a scholar infers from a SurveyUSA chart that twice as many pollsters were outside of the stated margin of sampling error as actually were, everyone is harmed, Mr. Panagakis says.

In practice: Mr. Panagakis would like SurveyUSA to divide all error numbers by 2.

This would create an “error on the candidate” instead of an “error on the margin of victory,” which is what SurevyUSA logs on its pollster compendium. An “error on the candidate” is what the National Council on Public Polls has used in its recent analyses of pollster performance.

SurveyUSA is happy to include an “error on the candidate” notation. Just to be clear: the error on the “margin of victory” and the “error on the candidate estimate” will always produce the same pollster ranking, in a two-candidate race. What this means is: Mr. Panagakis’ firm, Market Shares Corporation, would be the #1 ranked pollster, with the lowest mean error, no matter which of these two measures SurveyUSA uses. Take a look.

Going forward, SurveyUSA will incorporate a row that highlights the Panagakis Error on the Candidate measure when we produce tables such as the following, which we unveil here for the first time. There is a dedicated row for “Error on the Candidate.”

(Double-click on the image to enlarge it; click it again to make it legible.)

Suggestions for how to improve this “All Pollster x All Error Measures” grid, before SurveyUSA extends it to other election contests, are welcome.

Jay H Leve
Editor
SurveyUSA
editor@surveyusa.com