Voice of the Respondent: The Impact of Poor Respondent Experience
Author: Nicole Mitchell, Dynata
Survey respondents are the backbone of public opinion research. Their powerful insights help us to understand public sentiment, gauge public support for policies or initiatives, evaluate the impact of communication strategies, and aid in making informed decisions. However, as the demand for insights has increased, the willingness to participate in survey research among the population has continued to wane. The reduction of our most vital resource, the survey taker, comes not just from the volume of research they are asked to do, but poor survey experiences that deter survey takers from participating again.
Over our 40 years of experience, Dynata has been dedicated to doing research on research with the goal of improving the survey experience for respondents. Our research has consistently demonstrated that understanding the survey experience is essential to recruiting and retaining truly engaged online survey panelists. In 2022, nine of our US panelists aged 18 and older were recruited through an online survey to participate in one-on-one video interviews via Zoom to share their survey-taking experience with the industry. These video interviews were recorded and, with consent, were shared at various industry events, including the 2023 AAPOR Annual Conference.
Aligning with previous feedback from quantitative research conducted by Dynata, respondents mentioned poor experiences related to “long screeners”. Most surveys have a screening section that determines whether a participant qualifies for the main survey. However, long screeners terminate participants much further into the survey experience than is typical or acceptable. As noted by one of the interviewees, “I feel used, I feel, like, cheated,” explaining how they feel when disqualified after answering a large number of screening questions.
Another common theme that aligns with existing research on what causes poor respondent experience are long surveys. When asked to recall a long survey experience, an interviewee stated, “I thought I was at the end of this survey, and then here you are, you know, still asking me questions, so that would definitely be a time where I would say that was too long, because I’m like, oh, I’m finished, and then you proceed to ask me 20 more questions when I’m at like the 90% bar, and you squeeze in 15 questions and 10% of progress left”. Another interviewee stated that they “prefer the surveys of the shorter duration, you know, the 5-to-10-minute duration.”
Over two decades ago, Dynata asked panelists to tell us their ideal survey length. Almost 75% of participants indicated that the ideal survey length was between 2-10 minutes. Only 9% thought a survey over 15 minutes was ideal. Since then, we have repeatedly tested not just preference but performance, and consistently find that after 15 minutes, people’s attention and therefore the quality of the data, diminishes.
In addition to confirming previous feedback, the interviews uncovered coping strategies survey participants utilize to endure survey experiences that leave them physically and mentally exhausted. Multiple interviewees admitted to implementing coping strategies such as mindlessly clicking buttons with hopes of still qualifying or selecting less at “select all that apply” questions to push through the survey experience. These poor experiences turned highly engaged respondents into unengaged survey takers who rely on coping strategies that often result in poor quality data.
To ensure we are all good stewards of the research ecosystem, we need to elevate and consider survey participants as equal stakeholders in the research process. Providing a positive survey experience encourages positive engagement that not only contributes to better data quality but will also increase the likelihood of survey participation in the future.