WHAT
What questions were asked?
Find out the exact wording of poll questions to ensure that they are worded simply and clearly and are balanced and unbiased. A balanced question provides respondents with balanced answer options; if a reason or argument is provided in favor of one response, a reason for the alternative response should also be provided. Sometimes, particularly for questions with complex introductions or response options, it will be helpful to include the exact question wording in your story.
In what order were the questions asked?
Question order can impact the results. Could the questions asked prior to the question you’re interested in have influenced how respondents answered the following questions?
What other surveys have been done on this topic?
If other polls/surveys have been conducted on the same topic, it is good practice to compare results. If they were conducted at different times, the differences could represent a shift in public opinion (but note that different question wording and methods can lead to varying results). If they were conducted at the same time, see if the results are similar; if not, something could be up!
What is the margin of sampling error?
Probability sampling allows pollsters to calculate a margin of sampling error which is a measure of the possible range of approximation in the results due to sampling. Pollsters calculate the degree of certainty of results using a 95% confidence interval. That is, in 95 times out of 100, we expect that this confidence interval will include the true value of what we are trying to estimate. Generally, the more people who are interviewed in a poll, the smaller the error due to the size of the sample. But a larger sample isn’t necessarily better; other factors may be more important in judging the quality of any given survey.
What about election polling?
Like all polls, election polls represent a snapshot in time of public opinion, and they are not meant to be predictive of an outcome. Sampling error margins apply to the poll’s estimate of each candidate; a candidate usually needs to be ahead by 1.5–2 times the margin of sampling error for the lead to be statistically significant. Do not word poll results to sound like actual election results or say that a candidate is winning a race.
WHO
Who conducted the poll/survey?
Determine who conducted and paid for the poll/survey to evaluate credibility and whether they have a “dog in the fight” on issues their research is measuring attributes about. Reputable polling firms are more likely to provide the information you need to evaluate a survey’s methodology.
Are the results based on the answers of all the people interviewed, or only a subset?
Pay attention to whether particular questions were asked of all respondents or only of a certain subgroup. The margin of sampling error will vary for subgroups and can be much larger for small subgroups. If subgroups are very small, differences within the group should not be highlighted. If findings are among a subgroup, specify that in your story.
WHEN
When were the interviews conducted?
Pay attention to when the survey was conducted relative to recent events; opinions on gun control, for instance, may be different immediately following a mass shooting than they would be otherwise.
HOW
How was the survey conducted?
When the chance of selecting each person in the target population is known, the results of the survey are representative of the population. This is a probability sample.
When a sample is chosen without identifying the target population and without known probabilities of selecting each person, it is a non-probability sample. Error margins should not be reported for non-probability samples.
Both probability and non-probability approaches must rely on weighting to try to correct for any relevant ways in which the sample might be unrepresentative of the population.
It’s also important to note that there are different modes (phone, text, web, in-person) to conduct surveys, which can result in different results, known as “mode effects.”
How is a poll weighted?
Weighting accounts for unequal probabilities of selection and adjusts the poll data to ensure that the sample more accurately reflects the characteristics of the population from which it was drawn. Weighting can be used to adjust aspects of the sample that are mostly constant, such as demographics. Some demographic groups are more or less likely to respond to a survey, meaning they tend to be over-represented or under-represented in a given sample, so the weighting adjusts demographic variables to match known characteristics of the population obtained from an external source, such as the U.S. Census Bureau.
In vetting a survey, it’s important to examine whether the sources of weighting benchmarks are appropriately aligned to the sample that was drawn; for instance, weighting to recent national Census data is generally appropriate for a national sample of adults.
