Summary: Calculating response rates -  the number of eligible sample units that cooperate in a survey -- has historically been central to survey research in the United States because of the assumption that the larger the proportion of participating sample units, the more accurate the survey estimates.  Formulas for calculating rates are now standardized, but the relationship between response rates and survey quality has become much less clear.

Measuring Response Rates

Until recently, there were almost as many ways to calculate response rates as there were researchers. Response rates, cooperation rates, and completion rates were often treated as interchangeable in the literature.  In the early 1980s, the Council of American Survey Research Organizations (CASRO) made the first attempt to standardize the definition of a response rate, an effort completed in the late 1990s by AAPOR with the publication of Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys.

The 2008 edition of Standard Definitions clearly distinguishes between the response rate and the cooperation rate, covers household, telephone, mail, and Internet modes of administration, discusses the criteria for ineligibility, and specifies methods for calculating refusal and noncontact rates. As a result, response and nonresponse rates can now be successfully compared across surveys of different topics and organizations. In addition, these definitions and their widespread acceptance have resulted in a greater willingness of researchers to report low response rates.

Response Rates and Survey Quality

However, two factors have now undermined the role of the response rate as the primary arbiter of survey quality. Largely due to increasing refusals, response rates across all modes of survey administration have declined, in some cases precipitously. As a result, organizations have had to put additional effort into administration, thus making all types of surveys more costly. At the same time, studies that have compared survey estimates to benchmark data from the U.S. Census or very large governmental sample surveys have also questioned the positive association between response rates and quality. Furthermore, a growing emphasis on total survey error has caused methodologists to examine surveys—even those with acceptably high response rates--for evidence of nonresponse bias.

Results that show the least bias have turned out, in some cases, to come from surveys with less than optimal response rates. Experimental comparisons have also revealed few significant differences between estimates from surveys with low response rates and short field periods and surveys with high response rates and long field periods. (The difficulty of determining bias by comparing survey estimates to outside measurements, however, has led to ingenious strategies. One recent study developed an internal benchmark by using the 50/50 gender split of heterosexual, married couples to gauge the accuracy of survey estimates by gender among the respondents in six different surveys. )

There is currently no consensus about the factors that produce the disjuncture between response rates and survey quality. But the evidence does suggest several rules of thumb for consumers of survey reports and for researchers.

Researchers should always include in their survey reports the response rate, computed according to the appropriate AAPOR formula (see AAPOR Response Rate Calculator here - Excel) or another similar formula fully described. Furthermore, several other measures of quality should become part of reports, especially when a response rate is low. On their side, consumers of survey results should treat all response rates with skepticism, since these rates do not necessarily differentiate reliably between accurate and inaccurate data. Instead consumers should pay attention to other indicators of quality that are included in reports and on websites, such as insignificant levels of bias, low levels of missing data, and conformity with other research findings.

More Information

There is a rich literature on response rates and nonresponse bias. The most recent contributions can be found in the Special Issue of the Public Opinion Quarterly “Nonresponse Bias in Household Surveys” (2006). The monograph from the Second International Conference on Survey Nonresponse, Survey Nonresponse edited by Robert M. Groves, Don A. Dillman, John L. Eltinge, and Roderick A. J. Little, and published in 2002 is also a valuable reference.