Summary: Calculating response rates – the number of eligible sample units that cooperate in a survey — has historically been central to survey research in the United States because of the assumption that the larger the proportion of participating sample units, the more accurate the survey estimates. Formulas for calculating rates are now standardized, but the relationship between response rates and survey quality has become much less clear.
AAPOR has created two tools for calculating response rates:
Download Excel Response Rate Calculator
Until recently, there were almost as many ways to calculate response rates as there were researchers. Response rates, cooperation rates, and completion rates were often treated as interchangeable in the literature. In the early 1980s, the Council of American Survey Research Organizations (CASRO) made the first attempt to standardize the definition of a response rate, an effort completed in the late 1990s by AAPOR with the publication of Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys.
The Tenth Edition of Standard Definitions clearly distinguishes between the response rate and the cooperation rate, covers household, telephone, mail, and Internet modes of administration, discusses the criteria for ineligibility, and specifies methods for calculating refusal and noncontact rates. As a result, response and nonresponse rates can now be successfully compared across surveys of different topics and organizations. In addition, these definitions and their widespread acceptance have resulted in a greater willingness of researchers to report low response rates.
However, two factors have now undermined the role of the response rate as the primary arbiter of survey quality. Largely due to increasing refusals, response rates across all modes of survey administration have declined, in some cases precipitously. As a result, organizations have had to put additional effort into administration, thus making all types of surveys more costly. At the same time, studies that have compared survey estimates to benchmark data from the U.S. Census or very large governmental sample surveys have also questioned the positive association between response rates and quality. Furthermore, a growing emphasis on total survey error has caused methodologists to examine surveys – even those with acceptably high response rates–for evidence of nonresponse bias.
Results that show the least bias have turned out, in some cases, to come from surveys with less than optimal response rates. Experimental comparisons have also revealed few significant differences between estimates from surveys with low response rates and short field periods and surveys with high response rates and long field periods. (The difficulty of determining bias by comparing survey estimates to outside measurements, however, has led to ingenious strategies. One recent study developed an internal benchmark by using the 50/50 gender split of heterosexual, married couples to gauge the accuracy of survey estimates by gender among the respondents in six different surveys. )
There is currently no consensus about the factors that produce the disjuncture between response rates and survey quality. But the evidence does suggest several rules of thumb for consumers of survey reports and for researchers.
Researchers should always include in their survey reports the response rate, computed according to the appropriate AAPOR formula (Download AAPOR Response Rate Calculator here – Excel) or another similar formula fully described. Furthermore, several other measures of quality should become part of reports, especially when a response rate is low. On their side, consumers of survey results should treat all response rates with skepticism, since these rates do not necessarily differentiate reliably between accurate and inaccurate data. Instead consumers should pay attention to other indicators of quality that are included in reports and on websites, such as insignificant levels of bias, low levels of missing data, and conformity with other research findings.
There is a rich literature on response rates and nonresponse bias. The most recent contributions can be found in “The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis” (2008) and the Special Issue of the Public Opinion Quarterly “Nonresponse Bias in Household Surveys” (2006). The monograph from the Second International Conference on Survey Nonresponse, Survey Nonresponse edited by Robert M. Groves, Don A. Dillman, John L. Eltinge, and Roderick A. J. Little, and published in 2002 is also a valuable reference.
Members of the Standard Definitions Committee identified a number of relatively minor errors in the ‘Version 4.0 (DFRDD)’ tab. These errors affected only this tab; all other tabs were tested thoroughly. Any cooperation, refusal and contact rate calculations based on the previous calculators (Version 4.0 or prior) should be checked. Our below corrections do not impact response rate calculations. Note that we have also added combined cooperation, refusal, and contact rates to the DFRDD tab calculated on the same basis as the combined response-rates.
Cooperation Rate 1. The formula shown for COOP1 in cell A106 is incorrect. It is given as (I+INR)/(I+INR+R+(e2*(O+UO))). The correct formula for COOP1 is I / ((I + P) + R + O).
Cooperation Rate 2. The formula shown for COOP2 in cell A108 is incorrect. It was given as (I+P+INR)/(I+P+INR+R+(e2*(O+UO))). The correct formula for COOP2 is (I + P) / ((I +P) + R + O) (AAPOR 2016:63). The formulas of cells in row 108 are incorrect.
Cooperation Rate 3. The formula shown for COOP3 in cell A110 is incorrect. It is given as (I+INR)/(I+INR+R+(e2*UO)). The correct formula for COOP3 is I / ((I + P) + R) (AAPOR 2016:63). The formulas of cells in row 110 are incorrect. The formula equates to (I+INR)/((I+INR+P)+R+UO).
Cooperation Rate 4. The formula shown for COOP4 in cell A112 is incorrect. It is given as (I+INR)/(I+INR+R+(e2*UO)). The correct formula for COOP4 is (I + P) / ((I + P) + R) (AAPOR 2016:63). The formulas of cells in row 112 are incorrect. The formula equates to (I+P+INR)/((I+INR+P)+R+UO). The inclusion of INR, UO and 4.70 is incorrect.
Refusal Rate 1. The formula for REF1 in row 115 equates to (outcome code 3.211+R)/((I+P)+(R+NC+O)+(UH+UO)). The inclusion of outcome code 3.211 (no screener completed, residential and live contact made) is incorrect.
Refusal Rate 2. The formula shown for REF2 in cell A117 is incorrect. It is given as R/(I+P+R+NC+O+e(UH + UO)). It should be given as R / (I+P+R+NC+O+[e1*e2*UH]+[e1*UO]), following the RR3 formula (AAPOR 2016:69). The formula for REF2 in row 117 equates to (outcome code 3.211+R)/((I+P)+(R+NC+O)+(e2*(UH+UO))). The inclusion of outcome code 3.211 is incorrect and (e2*(UH+UO)) should be ((UH*e2*e1)+(UO*e1)), following the logic of RR2 (AAPOR 2016:69).
Refusal Rate 3. The formula for REF3 in row 119 equates to (outcome code 3.211+R)/((I+P)+(R+NC+O)+(e2*(UH+UO))). The inclusion of outcome code 3.211 is incorrect.
Contact Rate 1. The formula shown for CON1 in row 122 equates to ((I+P)+ outcome code 3.211+R+O)/((I+P)+R+NC+O+(UH+UO)). The inclusion of outcome code 3.211 is incorrect.
Contact Rate 2. The formula shown for CON2 in cell A124 is incorrect. It is given as (I+P+R+O) / (I+P+R+O+NC+e(UH+UO)). It should be given as (I+P+R+O) / (I+P+R+NC+O+[e1*e2*UH]+[e1*UO]). The formula for CON2 in row 122 equates to ((I+P)+3.211+R+O) / ((I+P)+R+NC+O+(UH+UO)). The inclusion of 3.211 is incorrect and (e2*(UH+UO)) is incorrect and should be ((UH*e2*e1)+(UO*e1), following the logic of RR3 (AAPOR 2016:69).
Contact Rate 3. The formula for CON3 in row 126 equates to ((I+P)+R+O) / ((I+P)+ outcome code 3.211+R+O+NC). The inclusion of outcome code 3.211 is incorrect.