Publications & Resources

New Task Force Report Released: Evaluating Survey Quality in Today’s Complex Environment


There now are more ways than ever to collect useful data. And while a well designed and carefully executed survey still can deliver valid and reliable results, methods and details matter more than ever.

It is in this context that AAPOR formed a task force to examine the current state of survey methods and provide guidance on the types of information survey practitioners and end users need in order to assess the quality and reliability of survey data. AAPOR has provided detailed treatment of many of these issues in earlier task force reports (Baker et al. 2013; Baker et al. 2010). But the survey research world is changing rapidly and it is critical that periodically we take a broader look at the changes afoot and offer guidance to practitioners and consumers on how to navigate the evolving landscape.

This document is comprised of a series of 17 questions that users of survey data should ask to help them make judgments about the validity of a survey’s results regardless of the method used. The answers are not definitive since the underlying issues are still much debated. Rather, they identify the types of design and implementation decisions that can have a significant impact on data outcomes. They are meant to alert the data user to potential sources of bias that might be present. In that sense they form a framework for assessing data quality from virtually any survey, while leaving the ultimate judgment about a survey’s usefulness to the data user in the context of the decision to be made or the phenomenon being studied.

Read full report…