Transparency Initiative Update
07/01/2020
The Transparency Initiative is pleased to report that it is closing in on 100 members as we enter our seventh year! New applications to the TI continue to arrive on a consistent basis, and those on the committee work hard to review new applicants as well as perform biennial evaluations. The commitment to methodological transparency shines in the work of the committee and TI members who approach their work from across the research spectrum. Committee members bring to the TI their experience in the worlds of non-profit and commercial research, as well as from government and academic settings. As research methodologies continue to evolve, the TI remains vigilant in ensuring that rigor and transparency defines the public face of its members regardless of mode or sampling frame.
All members are subject to reviews every two years. In 2019, the majority of those reviewed passed cleanly. Some (11 in total) passed with minor suggestions noted for improvement. Only two organizations were flagged for requiring major revisions in their publicly released disclosure statements. Reviewers found common mistakes in the following areas:
- Failure to describe a within-household selection procedure: It is incumbent on members to provide details on who was selected within a household. Was it the youngest male, for example? If so, members must be precise in their methodological statements.
- Incomplete descriptions of how weights were calculated, including the variables used and the sources of weighting parameters: It is not uncommon for members to do the right thing by noting the variables used for weighting, but failing to disclose the specific source of the weighting parameter such as the ACS, voter file, etc.
- Failure to identify the languages offered to survey participants: Members are required to disclose whether the research was conducted in English only or offered in multiple languages.
- Imprecise or misleading language around variance for non-probability data: Reports of non-probability samples should only provide measures of precision if they are accompanied by a detailed description of how the measure is specified and calculated. To avoid confusion, the TI suggests avoiding the use of “margin of error” terminology in conjunction with non-probability samples.
- Incomplete descriptions of noncoverage: Members must articulate if any portion of the target population is not covered by the sample frame, as well as estimate the size if possible. For example, a survey targeting all adults using a frame of listed addresses may only cover 97% of the population if it misses P.O. Boxes or newly constructed dwellings.
The TI will be embarking on its next round of biennial reviews in the coming months. TI committee members will review two publicly available reports in order to ensure that all required disclosure elements are included, and inform members of the review outcome upon completion.
TI committee members serve for two-year terms. It is an excellent way to become more plugged in to AAPOR, make professional and personal connections, and become more versed in the array of research approaches that make AAPOR such a vibrant and eclectic organization. Should you have any questions about the TI, please don’t hesitate to contact its co-chairs, Jennifer Benz or Krista Jenkins.