Disclosure Standards

Section III. of the AAPOR Code of Professional Ethics & Practice (April 2021) specifies:

III. Standards for Disclosure

Broadly defined, research on public opinion can be conducted using a variety of quantitative and qualitative methodologies, depending on the research questions to be addressed and available resources. Accordingly good professional practice imposes the obligation upon all public opinion and survey researchers to disclose sufficient information about how the research was conducted to allow for independent review and verification of research claims, regardless of the methodology used in the research. Full and complete disclosure for items listed in Section A will be made at the time results are released, either publicly or to a research client, as the case may be. As detailed below, the items listed in Section B, if not immediately available, will be released within 30 days of any request for such materials. If the results reported are based on multiple samples or multiple modes, the preceding items (as applicable) will be disclosed for each.

  1. Data Collection Strategy: Describe the data collection strategies employed (e.g. surveys, focus groups, content analyses).
  1. Who Sponsored the Research and Who Conducted It. Name the sponsor of the research and the party(ies) who conducted it. If the original source of funding is different than the sponsor, this source will also be disclosed.
  1. Measurement Tools/Instruments. Measurement tools include questionnaires with survey questions and response options, show cards, vignettes, or scripts used to guide discussions or interviews. The exact wording and presentation of any measurement tool from which results are reported as well as any preceding contextual information that might reasonably be expected to influence responses to the reported results and instructions to respondents or interviewers should be included. Also included are scripts used to guide discussions and semi-structured interviews and any instructions to researchers, interviewers, moderators, and participants in the research. Content analyses and ethnographic research will provide the scheme or guide used to categorize the data; researchers will also disclose if no formal scheme was used.
  1. Population Under Study. Survey and public opinion research can be conducted with many different populations including, but not limited to, the general public, voters, people working in particular sectors, blog postings, news broadcasts, an elected official’s social media feed. Researchers will be specific about the decision rules used to define the population when describing the study population, including location, age, other social or demographic characteristics (e.g., persons who access the internet), time (e.g., immigrants entering the US between 2015 and 2019). Content analyses will also include the unit of analysis (e.g., news article,  social media post) and the source of the data (e.g., Twitter, Lexis-Nexis).
  1. Method Used to Generate and Recruit the Sample. The description of the methods of sampling includes the sample design and methods used to contact or recruit research participants or collect units of analysis (content analysis).
    1. Explicitly state whether the sample comes from a frame selected using a probability-based methodology (meaning selecting potential participants with a known non-zero probability from a known frame) or if the sample was selected using non-probability methods (potential participants from opt-in, volunteer, or other sources).
    2. Probability-based sample specification should include a description of the sampling frame(s), list(s), or method(s).
      1. If a frame, list, or panel is used, the description should include the name of the supplier of the sample or list and nature of the list (e.g., registered voters in the state of Texas in 2018, pre-recruited panel or pool).
      2. If a frame, list, or panel is used, the description should include the coverage of the population, including describing any segment of the target population that is not covered by the design.
    3. For surveys, focus groups, or other forms of interviews, provide a clear indication of the method(s) by which participants were contacted, selected, recruited, intercepted, or otherwise contacted or encountered, along with any eligibility requirements and/or oversampling.
    4. Describe any use of quotas.
    5. Include the geographic location of data collection activities for any in-person research.
    6. For content analysis, detail the criteria or decision rules used to include or exclude elements of content and any approaches used to sample content. If a census of the target population of content was used, that will be explicitly stated.
    7. Provide details of any strategies used to help gain cooperation (e.g., advance contact, letters and scripts, compensation or incentives, refusal conversion contacts) whether for participation in a survey, group, panel, or for participation in a particular research project. Describe any compensation/incentives provided to research subjects and the method of delivery (debit card, gift card, cash).
  1. Method(s) and Mode(s) of Data Collection. Include a description of all mode(s) used to contact participants or collect data or information (e.g., CATI, CAPI, ACASI, IVR, mail, Web for survey; paper and pencil, audio or video recording for qualitative research, etc.) and the language(s) offered or included. For qualitative research such as in-depth interviews and focus groups, also include length of interviews or the focus group session.
  1. Dates of Data Collection. Disclose the dates of data collection (e.g., data collection from January 15 through March 10 of 2019). If this is a content analysis, include the dates of the content analyzed (e.g., social media posts between January 1 and 10, 2019).
  1. Sample Sizes (by sampling frame if more than one frame was used) and (if applicable) Discussion of the Precision of the Results.
    1. Provide sample sizes for each mode of data collection (for surveys include sample sizes for each frame, list, or panel used).
    2. For probability sample surveys, report estimates of sampling error (often described as “the margin of error”) and  discuss whether or not the reported sampling error or statistical analyses have been adjusted for the design effect due to weighting, clustering, or other factors.
    3. Reports of non-probability sample surveys will only provide measures of precision if they are defined and accompanied by a detailed description of how the underlying model was specified, its assumptions validated, and the measure(s) calculated.
    4. If content was analyzed using human coders, report the number of coders, whether inter-coder reliability estimates were calculated for any variables, and the resulting estimates.
  1. How the Data Were Weighted. Describe how the weights were calculated, including the variables used and the sources of the weighting parameters.
  1. How the Data Were Processed and Procedures to Ensure Data Quality. Describe validity checks, where applicable, including but not limited to whether the researcher added attention checks, logic checks, or excluded respondents who straight-lined or completed the survey under a certain time constraint, any screening of content for evidence that it originated from bots or fabricated profiles, re-contacts to confirm that the interview occurred or to verify respondent’s identity or both, and measures to prevent respondents from completing the survey more than once. Any data imputation or other data exclusions or replacement will also be discussed. Researchers will provide information about whether any coding was done by software or human coders (or both); if automated coding was done, name the software and specify the parameters or decision rules that were used.
  1. A General Statement Acknowledging Limitations of the Design and Data Collection. All research has limitations and researchers will include a general statement acknowledging the unmeasured error associated with all forms of public opinion research.
  1. Procedures for managing the membership, participation, and attrition of the panel, if a pool, panel, or access panel was used. This should be disclosed for both probability and non-probability surveys relying on recruited panels of participants.
  2. Methods of interviewer or coder training and details of supervision and monitoring of interviewers or human coders. If machine coding was conducted, include a description of the machine learning involved in the coding.
  3. Details about screening procedures, including any screening for other surveys or data collection that would have made sample or selected members ineligible for the current data collection (e.g., survey, focus group, interview) will be disclosed (e.g., in the case of online surveys if a router was used).
  4. Any relevant stimuli, such as visual or sensory exhibits or show cards. In the case of surveys conducted via self-administered computer-assisted interviewing, providing the relevant screen shot(s) is strongly encouraged, though not required.
  5. Summaries of the disposition of study-specific sample records so that response rates for probability samples and participation rates for non-probability samples can be computed. If response or cooperation rates are reported, they will be computed according to AAPOR Standard Definitions. If dispositions cannot be provided, explain the reason(s) why they cannot be disclosed, and this will be mentioned as a limitation of the study.
  6. The unweighted sample size(s) on which one or more reported subgroup estimates are based.
  7. Specifications adequate for replication of indices or statistical modeling included in research reports.

Reflecting the fundamental goals of transparency and replicability, AAPOR members share the expectation that access to datasets and related documentation will be provided to allow for independent review and verification of research claims upon request. In order to protect the privacy of individual respondents, such datasets will be de-identified to remove variables that can reasonably be expected to identify a respondent. Datasets may be held without release for a period of up to one year after findings are publicly released to allow full opportunity for primary analysis. Those who commission publicly disseminated research have an obligation to disclose the rationale for why eventual public release or access to the datasets is not possible, if that is the case.

If any of our work becomes the subject of a formal investigation of an alleged violation of this Code, undertaken with the approval of the AAPOR Executive Council, we will provide additional information on the research study in such detail that a fellow researcher would be able to conduct a professional evaluation of the study.

Disclosure FAQs

Yes. The disclosure requirement applies only to surveys for which results are released to the public.

Yes. The AAPOR Code represents ethical standards that apply to all public opinion researchers. If a complaint of non-disclosure is received, AAPOR can and will censure researchers (and research companies) who fail to disclose their methods as specified in the AAPOR Code, whether or not the individuals are AAPOR members.

No. AAPOR recognizes that reports about research take many different forms, and that inclusion of all the methods information is not always appropriate. The Code prescribes that researchers will include this information in “any report of research results,” or that they will “make [it] available when that report is released.” In most cases, this means that the researcher will prepare a detailed methodological appendix as a part of a complete research report. Summaries and shorter forms of reports can then refer to the full report as a source for the details of the method, and when such reference is made, the report is taken as “including” the details. The description of method can also be prepared as a separate memo or document, which can be made available in printed or electronic form, for example, by posting on a website. Good practice dictates that public research reports should indicate, whenever practical, where the description of method can be obtained. In any case, the researcher is required to disclose the information listed in the Standards for Minimal Disclosure upon request from any member of the public.

Not necessarily. You can meet the requirement for release of question wording (§III.2) by disclosing the wording of only those questions for which results were released, along with the text of any preceding instruction, explanation to the interviewer or respondents, or previous questions that might reasonably be expected to affect the response.

Not necessarily. Note, however, that you do have to release the wording of introductions and instructions that could affect the responses to released questions. Even if these materials mention or imply the existence of the unreleased questions or items, they would still need to be disclosed if they could reasonably affect any of the released questions. Also, if the presence in the survey instrument of the unreleased questions could reasonably be expected to influence the responses to the questions for which results are released, then the existence of those questions must be acknowledged and the text of those questions ought to be released as well, in the interest of full disclosure. In general, it is best to place questions for which you do not intend to release results so that they follow the questions designed for release, so that such effects will be minimized.

All research firms, organizations and individuals are obligated to make the minimally required disclosures, upon request, for any publicly released survey results. It is permissible to describe proprietary methods in general terms, without necessarily providing all the details that another researcher would need to fully replicate the procedure. For example: “The results for this election poll were weighted according to the respondent’s likelihood of voting, using a proprietary formula based on several questions about voting intention and past voting.”

Yes, if the posting is on a publicly accessible website. If you are not willing to make these disclosures, then don’t make the results publicly available. Save these teaser results for display in your private or individually targeted sales pitches.

Under these circumstances, the obligation to disclose would be upon the research sponsor (i.e., your client), assuming the sponsor has released the survey’s results. Refer requests for disclosure to the research sponsor. The Code addresses this issue specifically at §II.A.3:

We shall inform those for whom we conduct publicly released surveys that AAPOR standards require members to release minimal information about such surveys, and we shall make all reasonable efforts to encourage clients to subscribe to our standards for minimal disclosure in their releases.

If key personnel on the project are AAPOR members, they can avoid any misunderstanding on this point by including language similar to the following in the contract with the client (assuming the client is the Sponsor and the AAPOR members’ firm is in the Consultant role):

Applicability of AAPOR Code to publicly released results. The Sponsor recognizes that the Consultant’s key personnel are members of the American Association for Public Opinion Research (AAPOR) and are bound to adhere to its Code of Ethics and Professional Practicies. Nothing in this contract shall prevent the Consultant from fulfilling its obligation to ensure disclosure by the Sponsor, or the Consultant, upon request from any member of the public, minimally required information about the survey methods as required by the AAPOR Code of Ethics and Professional Practices, as amended on April, 2021. The requirements of the AAPOR Standards for Minimal Disclosure (Section III of the Code) will apply only to those survey questions, if any, for which statistical results are publicly released by the Sponsor or the Consultant.

Got a question about disclosure?

Contact the AAPOR Standards Chair or Associate Standards Chair for assistance. As questions and answers accumulate, they may be added to this document.

Disclosure FAQ 4/2021
Code as revised 4/2021

Survey Disclosure Checklist

Participation in AAPOR’s Transparency Initiative involves willingness to provide the items listed below in any reports of survey research results that are publicly released, or to make them publicly available, preferably on the organization’s web site, immediately upon release of that report. These items correspond to the first part of Section III.A of AAPOR’s Code of Professional Ethics and Practices, which was revised in April 2021.

  1. Data Collection Strategy
  2. Who Sponsored the Research and Who Conducted It
  3. Measurement Tools/Instruments
  4. Population Under Study
  5. Method Used to Generate and Recruit the Sample
  6. Method(s) and Mode(s) of Data Collection
  7. Dates of Data Collection
  8. Sample Sizes (by sampling frame if more than one frame was used) and (if applicable) Discussion of the Precision of the Results
  9. How the Data Were Weighted
  10. How the Data Were Processed and Procedures to Ensure Data Quality
  11. A General Statement Acknowledging Limitations of the Design and Data Collection

For more detailed information on these required items for disclosure, please review the full checklist.

Approved for posting by AAPOR Council, April, 2021