Statements

AAPOR Response to New York Times / CBS News poll

08/01/2014

The Critical Role of Transparency & Standards in Today’s World of Polling and Opinion Research

The world of polling and opinion research is indeed in the midst of significant change, offering opportunities for new insights into public thinking but also with the potential for us to lose our bearings in terms of what is and is not accurate information. Recent actions involving the removal of meaningful standards of publication for polling data by at least one leading media outlet and the publication of polling stories using opt-in Internet survey data have raised concerns among many in the field that we are witnessing some of the potential dangers of rushing to embrace new approaches without an adequate understanding of the limits of these nascent methodologies. The American Association for Public Opinion Research (AAPOR), the leading association of public opinion and survey researchers, recognizes that the ways in which public opinion is formed, expressed, conceptualized and measured continue to grow and change. We embrace rigorous empirical testing of these new approaches and methodologies and encourage assessments of their viability for measurement and insight. It is essential, however, that the use of any new methods be conducted within a strong framework of transparency, full disclosure and explicit standards. The absence of these foundational necessities removes the ability for any of us to understand the quality and validity of the information reported.

AAPOR has for decades worked to encourage disclosure of methods and the use of objective standards in the assessment and reporting of public opinion and polling data. This allows others to understand and evaluate for themselves how information is collected and hence the likely veracity of the information being reported. Transparency in approach is even more critical when new or experimental approaches are being tried. This week, the New York Times and CBS News published a story using, in part, information from a non-probability, opt-in survey sparking concern among many in the polling community. In general, these methods have little grounding in theory and the results can vary widely based on the particular method used.  While little information about the methodology accompanied the story, a high level overview of the methodology was posted subsequently on the polling vendor’s website. Unfortunately, due perhaps in part to the novelty of the approach used, many of the details required to honestly assess the methodology remain undisclosed. This may be an isolated incident with the Times / CBS News providing more information on this effort in the coming weeks. If not, it is a disappointing precedent being set by two of our leading media institutions.

AAPOR is committed to promoting full disclosure in any and all methods used to collect and produce inferences from public opinion data. The association is launching an industry-wide Transparency Initiative later this year to help educate the industry in this area and to publicly recognize those organizations that do routinely disclose their methods. The Times and CBS News were, in fact, two of the first organizations to voice support for the AAPOR Transparency Initiative when it was first announced, support which we acknowledge and for which we are grateful.

Standards of quality are also a necessary component in this world of new approaches to ensure that information relayed about public attitudes and behaviors meets at least a minimum threshold of confidence that the data are accurate and the insights published do, in fact, mirror reality. Until this week, the Times maintained and published a set of rigorous standards to guide the determination about when polling data could (and could not) be used in a story. Those detailed standards were summarily removed and replaced with a statement indicating that the old standards were undergoing review and that “individual decisions about which poll meets Times standards and specifically how they should be used” would guide decisions in the interim. This means no standards are currently in place.  It is unclear why the decision was made to pull the existing standards before the new ones are developed, vetted and published. Yes, all responsible institutions need to review their standards periodically, making appropriate changes as technologies and methodologies change. However, standards need to be in place at all times precisely to avoid the “we know it when we see it (or worse yet, ‘prefer it’)” approach, which often gives expediency and flash far greater weight than confidence and veracity.

A case-in-point, earlier this year the Times published a poll-based story on undocumented immigrants (“Young Undocumented Immigrants Growing Disenchanted with Both Parties, Study Finds,” 5/21/14) and claimed to report on new insights into the views of younger immigrants. The study results were published even though the methodology used was clearly in violation of the Times own standards at the time. A quick review of the limited information on how the survey was conducted – utilizing an opt-in Internet panel heavily skewed towards more educated and affluent immigrants —  rendered the “findings” in the piece highly questionable, if not outright incorrect. The result was greater misunderstanding than clarity on this important issue – and a black-eye, not just for the Times, but the public opinion industry more generally. Using information from polls which are not conducted with scientific rigor in effect sets a new — lower — standard for the types of information that other news outlets may now seek to report.

As an association comprised of members from many sectors and leading opinion and polling organizations, AAPOR is committed to a fundamental belief in a scientific approach to the generation of data and their systematic interpretation. As the approaches for collecting and interpreting public opinion data – in all its forms — continue to multiply and diversify, AAPOR is committed to exerting leadership in maintaining data quality regardless of the methodologies being employed. To this end, we strongly encourage all polling outlets to proceed cautiously in the use of new approaches to measuring public opinion, such as those using non-probability sample methods, particularly when these data are used in news or election reporting or public policymaking. An AAPOR task force published an overview of some of these types of methods, including a discussion of when they may or may not be suitable for a particular purpose. The report is freely available to all. Additionally, we strongly urge the disclosure of the methods used as well as the application of explicit standards by which to help assess what data should (and should not) be put forth.

We applaud the many journalists, decisionmakers, and researchers who support and practice transparency, full disclosure of methods, and the use of quality standards – especially in times of changing methods and approaches. We welcome the opportunity to work with those interested in learning more about developing and implementing disclosure rules, standards of practice, and best practices in the use of new and more traditional forms of opinion-related data.

Michael Link, Ph.D.
President, AAPOR


August 1, 2014
Scott Keeter, AAPOR past president and the director of survey research at the Pew Research Center, provides additional thoughts on the implications and cautions of using non-probability methods to collect election polls. Read the article here.