This is an archive of the old Stones Cry Out site. For the current site, click here.

« How Sharper Than a Serpent's Tooth | Main | What's Wrong With Kansas? »

March 15, 2005

New Exit Poll Paper

The National Research Commission on Elections and Voting recently released a working paper entitled, “A Review of Recent Controversies Concerning the 2004 Presidential Election Exit Polls.”

The paper is largely a compilation of known information procured from a variety of internet resources, evinced by the 50 URLs included in the footnotes to the 18-page report. Several blogs were cited in the report, but not as frequently and with as much authority as Mystery Pollster. Overall, I found the paper to be an interesting and informative read.

The biggest flaw of the paper, though, is that the authors give the January 19, 2005 NEP Report prepared by Edison Media/Mitofsky International too much credit. The authors note that the NEP Report provides an “unprecedented amount of information…much more than has ever been released before” (p. 9) but concede that one of the problems with the post-survey evaluations is that “some information is essentially unknowable” (p. 8). While these statements are true, the NEP Report could have taken some of the analysis a step further.

Perhaps the most prominent critical response to the NEP Report is a paper written by a group of highly credentialed academics writing for US Count Votes. Although the US Count Votes paper attacked the NEP Report on a variety of fronts, the most serious charge in the paper was regarding the NEP Report’s handling of WPE by voting method. The NEP Report stated,

WPE in precincts with any type of automated voting system is higher than the average error in paper ballot precincts. These errors are not necessarily a function of the voting equipment. They appear to be a function of the equipment’s location and the voters’ responses to the exit poll at precincts that used this equipment (p. 40).
The authors of the US Count Votes paper zero in on that first statement and reproduce a table showing that the median WPE for precincts with paper ballots was -0.9, but anywhere from -5.5 to -10.3 for all other voting methods, including optical scan, punch cards, touch screen, and mechanical voting. The implication being that automated voting technology could have been tampered with, accounting for the substantially different WPE when compared to WPE in precincts with paper ballots.

However, if you look at the mean and median WPE by urban vs. rural, the differences appear to vanish. The NEP Report explains,

The low value of the WPE in paper ballot precincts may be due to the location of those precincts in rural areas, which had a lower WPE than other places (p. 40).
This apparent case of Simpson’s Paradox seems ripe for more rigorous statistical analysis and unlike the Commission paper, the US Count Votes paper makes the call:
The Edison/Mitofsky Report does not report having done an ANOVA (analysis of variance) of voting machine type that might confirm their claim that there is no difference between precincts using different types of voting machines (p 4-5).
ANOVA would seem to be a no-brainer here and why the NEP Report does not include the results of this test is baffling. If a significant difference by voting method is found in the aggregate, the data can be re-aggregated by rural v. urban and retested. If these differences are not significant, then report this fact and explain what it is about the data that could cause the discrepant significance findings. If the discrepancies are significant, then some other explanation of the discrepancy should be pursued.

This leads me back to my claim that the Commission paper is too soft on the NEP Report. First, the NEP Report should have included results of an ANOVA test of WPE by vote method. If not, they should make the data available so it can be independently tested. The concluding paragraph of the Commission paper reads as follows:

The information on the exit poll methodology is still being consumed by independent analysts, and there are now calls for the release of raw and supplementary data from sample precincts. This would include contextual data about the vote history in those areas as well as information about the interviewers. This is unlikely to happen, and for justifiable reasons. Such information would be too politically sensitive in that disclosure of the sample sites could subject the exit poll interviewing to manipulation by political organizations and interest groups on Election Day if the same sites are always chosen (p. 13).
While I can sympathize with these concerns, an ANOVA of the WPE by voting machine would not require this type of data. To run the ANOVA, all that is needed is a spreadsheet with four columns: 1) dummy precinct ID (doesn’t have to be real); 2) WPE for the precinct; 3) type of voting method used in the precinct; and 4) precinct population. SPSS can calculate the rest.

Posted by Rick at March 15, 2005 04:20 PM

Trackback Pings

Listed below are links to weblogs that reference New Exit Poll Paper:

» Pounding My Head Against a Rock? from Wallo World
Drew from Stones Cry Out suggests that I'm thinking far too deeply in my recent post about the final installment of George Lucas' Star Wars saga. He's undoubtedly right; it's my curse, actually: I think too hard about almost everything. As a resu... [Read More]

Tracked on March 15, 2005 06:58 PM