For a long time, survey researchers have needed more comprehensive and reliable diagnostic tools to understand the components of total survey error. Some of those components, such as margin of sampling error, are relatively easily calculated and familiar to many who use survey research. Other components, such as the influence of question-wording on responses, are more difficult to ascertain. Groves (1989) catalogues error into three other major potential areas in which it can occur in sample surveys. One is coverage, where error can result if some members of the population under study do not have a known nonzero chance of being included in the sample. Another is measurement effect, such as when the instrument or items on the instrument are constructed in such a way to produce unreliable or invalid data. The third is nonresponse effect, where nonrespondents in the sample that researchers originally drew differ from respondents in ways that are germane to the objectives of the survey.
Defining final disposition codes and calculating survey outcome rates is the topic for the Standard Definitions report. Often it is assumed — correctly or not — that the lower the response rate, the more question there is about the validity of the sample. Although response rate information alone is not sufficient for determining how much nonresponse error exists in a survey, or even whether it exists, calculating the rates is a critical first step to understanding the presence of this component of potential survey error. By knowing the disposition of every element drawn in a survey sample, researchers can assess whether their sample might contain nonresponse error and the potential reasons for that error.
With this report AAPOR offers a tool that can be used as a guide to one important aspect of a survey’s quality. It is a comprehensive, well-delineated way of describing the final disposition of cases and calculating outcome rates for surveys conducted by telephone (landline and cell), for personal interviews in a sample of households, for mail surveys of specifically named persons (i.e., a survey in which named persons are the sampled elements), and for Web surveys.
AAPOR urges all practitioners to use these standardized sample disposition codes in all reports of survey methods, no matter if the project is proprietary work for private sector clients or a public, government or academic survey. This will enable researchers to find common ground on which to compare the outcome rates for different surveys.