Not so fast: The Polls and the Iowa Caucuses
The nation’s largest organization of survey researchers, the American Association for Public Opinion Research, urges pundits and journalists not to rush to judgment on the performance of polls in the aftermath of the Iowa Democratic Caucuses.
As the first official event of the 2020 presidential nomination campaign, the Iowa caucuses arrive after dozens and dozens of published polls on the Democratic race. As caucuses, the Iowa event poses challenges that differ from comparing results of polls to those of primary elections.
“The Democratic caucuses are not ‘just like’ a primary election. The results of the caucuses are more complex than a simple vote count,” said Nora Cate Schaeffer, AAPOR President.
The Iowa Democratic Party and the Iowa Republican Party hold separate precinct caucuses on Monday evening, Feb. 3, 2020. Many polls have focused on the Democratic caucuses, while few polls have been published on the GOP caucuses.
- AAPOR implores journalists to wait for final caucus results before analyzing polls’ performance. Because the caucuses are run by the political parties, not by state governments under state laws, the reporting of results is handled by the parties. This means there may be varying reports of results, especially early in the evening.
“Judging how well the pre-caucus polls of Iowa Democrats have performed must wait until the final caucus results are in,” says Schaeffer.
The parties have not always finished reporting results on caucus night. In 2012, the partial Republican caucus results showed Mitt Romney winning, but final results gave the victory to Rick Santorum by a tiny margin. In 1988, Democratic contender Paul Simon claimed he won the caucuses, but the party never published a complete count that could confirm his claim.
- Analyses of poll performance must take into consideration the fundamental uncertainty in all sample surveys, usually stated as the margin of sampling error. Failure to do so could result in a flawed analysis. This year, margins of error are critical since Iowa polls show Democrats are splintered in their preferences, with some candidates closely bunched and no candidate with more than 25% among potential caucus-goers.
This means that apparent differences among candidates may be within the margin of error. For example, in a January 2-8, 2020 CNN/ Des Moines Register/Mediacom Iowa Poll, four candidates were closely bunched: Bernie Sanders (20%), Elizabeth Warren (17%), Pete Buttigieg (16%) and Joe Biden (15%). The poll had a sampling error margin of 3.7 percentage points. That margin of error means no candidate is clearly the leader.
- Do not cherry-pick which caucus results to compare to polls, AAPOR advises. The state party plans to report at least three separate caucus numbers, which could vary from each other in terms of the leading candidate and candidate order. Only the statewide initial presidential preference number should be compared to pre-caucus polls. All other results are calculated after application of state party rules, which can result in some candidates showing no supporters at a given level. (For a visual explanation of the caucus rules, see https://www.politico.com/interactives/2020/iowa-caucus-how-they-work/).
“Comparing a pre-caucus poll to the wrong caucus results may result in an inaccurate conclusion,” said Schaeffer.
Performance of Polls in Recent Elections
Pundits and journalists have paid increasing attention to how closely poll results match election results in the last few decades, even as some in the polling industry have debated the value of giving substantial attention to how closely poll results have matched election returns.
Major studies in the last four years show polling is as accurate as it has ever been. The national presidential polls in the 2016 general election were quite accurate by historical standards, despite frequent mistaken claims to the contrary. In fact, they were among the most accurate in estimating the national popular vote since 1936. (See An Evaluation of 2016 Election Polls in the U.S.) State-level polling was not as accurate in the 2016 general election (notably in Pennsylvania, Michigan and Wisconsin), but it was reasonably accurate in the 2016 primary elections.
The 2016 presidential primary polls generally performed on par relative to past elections. The vast majority of primary polls predicted the right winner, with the prediction widely off the mark in only a few states. In short, the primary polls held their own in 2016.
—An Evaluation of 2016 Election Polls in the U.S., AAPOR Ad Hoc Committee On 2016 Election Polling, page 16.
In the 2018 elections, the state-level general election polls for Senate and Governor were the most accurate in off-year elections since 2006.
Contact: Eric Bailey, [email protected]