Ethical Issues in Linking Non-Survey Data to Survey Data: A Practical Guide
07/23/2025
07/23/2025
David Dutwin
May 2019
“Many of you know me primarily as a methodologist. But in fact, my path to AAPOR had nothing to do with methodology. My early papers, in fact, wholly either provided criticism of, or underscored the critical value of, public opinion and public opinion polls.
And so in some respects, this Presidential Address is for me, completes a full circle of thought and passion I have for AAPOR, for today I would like to discuss matters pertaining to the need to reconsider, strengthen, and advance the mission of survey research in democracy.
Historically, there has been much to say on the role of public opinion in democracy. George Gallup summarized the role of polls quite succinctly when he said, “Without polls, [elites] would be guided only by letters to congressmen, the lobbying of pressure groups, and the reports of political henchmen.”
Further, Democratic theory notes the critical, if not the pivotal role, of public opinion and democratic practice. Storied political scientist V.O Key said: “The poll furnishes a means for the deflation of the extreme claims of pressure groups and for the testing of their extravagant claims of public sentiment in support of their demands.”
Furthermore, surveys provide a critical check and balance to other claims of what the American public demands in terms of policies and their government. Without polls, it would be all that much harder to verify and combat claims of public sentiment made by politicians, elites, lobbyists, and interest groups. [“No policy that does not rest upon some public opinion can be permanently maintained.”- Abe Lincoln; “Public opinion is a thermometer a monarch should constantly consult” – Napoleon]
It is sometimes asked whether leaders do consult polls and whether polls have any impact of policy. The relationship here is complex, but time and again researchers have found a meaningful and significant effect of public opinion, typically as measured by polling, on public policy. As one example, Page and Shapiro explored trends in American public opinion from the 1930s to the 1980s and found no less than 231 different changes in public policy following shifts in public opinion.
And certainly, in modern times around the world, there is recognition that the loss of public opinion would be, indeed, the loss of democracy itself. [“Where there is no public opinion, there is likely to be bad government, which sooner or later, becomes autocratic government.” – Willian Lyon Mackenzie King]
And yet, not all agree. Some twist polling to be a tool that works against democratic principles. [“The polls are just being used as another tool for voter suppression.” – Rush Limbaugh]
And certainly, public opinion itself is imperfect, filled with non-attitudes, the will of the crowd, and can often lead to tyranny of the majority, as Jon Stewart nicely pointed out. [“You have to remember one thing about the will of the people: It wasn’t that long ago that we were swept away by the Macarena.” – Jon Stewart]
If these later quotes were the extent of criticism on the role of public opinion and survey research in liberal democracy, I would not be up here today discussing what soon follows in this address. Unfortunately, however, we live a world in which many of the institutions of democracy and society are under attack.
It is important to start by recognizing that AAPOR is a scientific organization. Whether you are a quantitative or qualitative researcher, a political pollster or developer of official statistics, a sociologist or a political scientist, someone who works for a commercial entity or nonprofit, we are all survey scientists, and we come together as a great community of scientists within AAPOR, no matter our differences.
And so we, AAPOR, should be as concerned as any other scientific community regarding the current environment where science is under attack, devalued, and delegitimized. It is estimated that since the 2016 election, the federal policy has moved to censor, or misrepresent, or curtail and suppress scientific data and discoveries over 200 times, according to the Sabin Center at Columbia University. Not only is this a concern to AAPOR as a community of scientists, but we should be concerned as well on the impact of these attacks on public opinion itself.
Just as concerning is the attack on democratic information, in general. Farrell and Schneier argue that there are two key types of knowledge in democracy, common and contested. And while we should be free to argue and disagree with policy choices, our pick of democratic leaders, and even many of the rules and mores that guide us as a society, what is called contested knowledge, what cannot be up for debate is the common knowledge of democracy, for example, the legitimacy of the electoral process itself, or the validity of data attained by the Census, or even more so, I would argue, that public opinion does not tell us what the public thinks.
As the many quotes I provided earlier attest to, democracy is dependent upon a reliable and nonideological measure of the will of the people. For more than a half century and beyond, survey research been the principal and predominant vehicle by which such knowledge is generated.
And yet, we are on that doorstep where common knowledge is becoming contested. We are entering, I fear, a new phase of poll delegitimization. I am not here to advocate any political ideology and it is critical for pollsters to remain within the confines of science. Yet there has been a sea change in how polls are discussed by the current administration. To constantly call out polls for being fake is to delegitimize public opinion itself and is a threat to our profession.
Worse still, many call out polls as mere propaganda (see Joondeph, 2018). Such statements are more so a direct attack on our science, our field, and frankly, the entire AAPOR community. And yet even worse is for anyone to actually rig poll results. Perhaps nothing may undermine the science and legitimacy of polling more.
More pernicious still, we are on the precipice of an age where faking anything is possible. The technology now exists to fake actual videos of politicians, or anyone for that matter, and to create realistic false statements. The faking of poll results is merely in lockstep with these developments.
There are, perhaps, many of you in this room who don’t directly connect with this. You do not do political polling. You do government statistics. Sociology. Research on health, on education, or consumer research. But we must all realize that polling is the tip of the spear. It is what the ordinary citizen sees of our trade and our science. As Andy Kohut once noted, it represents all of survey research. [Political polling is the “most visible expression of the validity of the survey research method.“ – Andrew Kohut]
With attacks on science at an all-time high in the modern age, including attacks on the science of surveys; with denigration of common knowledge, the glue that holds democracy together, including denunciation on the reliability of official statistics; with slander on polling that goes beyond deliberation on the validity of good methods but rather attacks good methods as junk, as propaganda, and as fake news; and worse of all, a future that, by all indications, will if anything include the increased frequency of fake polls, and fake data, well, what are we, AAPOR, to do?
We must respond. We must react. And, we must speak out. What does this mean, exactly? First, AAPOR must be able to respond. Specifically, AAPOR must have vehicles and avenues of communication and the tools by which it can communicate. Second, AAPOR must know how to respond. That is to say, AAPOR must have effective and timely means of responding. We are in an every minute of the day news cycle. AAPOR must adapt to this environment and maximize its impact by speaking effectively within this communication environment. And third, AAPOR must, quite simply, have the willpower to respond. AAPOR is a fabulous member organization, providing great service to its members in terms of education, a code of ethics, guidelines for best practices and promotions of transparency and diversity in the field of survey research. But we have to do more. We have to learn to professionalize our communication and advocate for our members and our field. There are no such thing as sidelines anymore. We must do our part to defend survey science, polling, and the very role of public opinion in a functioning democracy.
This might seem to many of you like a fresh idea, and bold new step for AAPOR. But in fact, there has been a common and consistent call for improved communication abilities, communicative outreach, and advocacy by many past Presidents, from Diane Colasanto to Nancy Belden to Andy Kohut.
Past President Frank Newport for example was and is a strong supporter of the role of public opinion in democracy, underscoring in his Presidential address that quote, “the collective views of the people…are absolutely vital to the decision-making that ultimately affects them.” He argued in his Presidential address that AAPOR must protect the role of public opinion in society.
A number of Past Presidents have rightly noted that AAPOR must recognize the central role of journalists in this regard, who have the power to frame polling as a positive or negative influence on society. President Nancy Mathiowetz rightly pointed out that AAPOR must play a role in, and even financially support, endeavors to guarantee that journalists’ support AAPOR’s position on the role of polling in society and journalists’ treatment of polls. And Nancy’s vision, in fact, launched relationship with Poynter in building a number of resources for journalist education of polling.
Past President Scott Keeter also noted the need for AAPOR to do everything it can to promote public opinion research. He said that “we all do everything we can to defend high-quality survey research, its producers, and those who distribute it.” But at the same time, Scott noted clearly that, unfortunately, “At AAPOR we are fighting a mostly defensive war.”
And finally, Past President Cliff Zukin got straight to the point in his Presidential address, noting that, quote “AAPOR needs to increase its organizational capacity to respond and communicate, both internally and externally. We need to communicate our positions and values to the outside world, and we need to diffuse ideas more quickly within our profession.”
AAPOR is a wonderful organization, and in my biased opinion, the best professional organization I know. How have we responded to the call of past Presidents? I would say, we responded with vigor, with energy, and with passion. But we are but a volunteer organization of social scientists. And so, we make task forces. We write reports. These reports are well researched, well written, and at the same time, I would argue, do not work effectively to create impact in the modern communication environment.
We have taken one small step to ameliorate this, with the report on polling in the 2016 election, which was publicly released via a quite successful live Facebook video event. But we can still do better. We need to be more timely for one, as that event occurred 177 days after the election, when far fewer people were listening, and the narrative was largely already written. And we need to find ways to make such events have greater reach and impact. And of course, we need more than just one event every four years.
I have been proud to have been a part of, and even be the chair of, a number of excellent task force reports. But we cannot, I submit, continue to respond only with task force reports. AAPOR is comprised of the greatest survey researchers in the world. But it is not comprised of professional communication strategists, plain and simple. We need help, and we need professional help.
In the growth of many organizations, there comes a time when the next step must be taken. The ASA many years ago, for example, hired a full time strategic communications firm. Other organizations, including the NCA, APSA, and others, chose instead to hire their own full time professional communication strategist.
AAPOR has desired to better advocate for itself for decades. We recognize that we have to get into the fight, that there are again no more things as sidelines. And we have put forward a commendable effort in this regard, building educational resources for journalists, and writing excellent reports on elections, best practices, sugging and frugging, data falsification, and other issues. But we need to do more, and in the context of the world outside of us, we need to speak a language that resonates with journalists, political elites, and perhaps most importantly the public.
I want to stop right here and make it clear, that the return on investment on such efforts is not going to be quick. And the goal here is not to improve response rates, though I would like that very much! No, it is not likely that any efforts in any near term reverses trends in nonresponse.
It may very well be that our efforts only slow or at best stop the decline. But that would be an important development. The Washington Post says that democracy dies in darkness. If I may, I would argue that AAPOR must say, democracy dies in silence, when the vehicle for public opinion, surveys, has been twisted to be distrusted by the very people who need it most, ordinary citizens. For the most part, AAPOR has been silent. We can be silent no more.
This year, Executive Council has deliberated the issues outlined in this address, and we have chosen to act. The road will be long, and at this time, I cannot tell you where it will lead. But I can tell you our intentions and aspirations. We have begun to execute a 5 point plan that I present here to you.
First, AAPOR Executive Council developed and released a request for proposals for professional strategic communication services. Five highly regarded firms responded. After careful deliberation and in person meetings with the best of these firms, we have chosen Stanton Communications to help AAPOR become a more professionalized association. Our goals in the short term are as follows.
We desire to become more nimble and effective at responding to attacks on polls, with key AAPOR members serving as spokespersons when needed, but only after professional development of the messages they will promulgate, approved by Council, and professionalized by the firm. Stanton brings with it a considerable distribution network of journalists and media outlets. AAPOR, through its professional management firm Kellen, has access to audio and video services of the National Press Club, and will utilize these services when needed to respond to attacks on polls, and for other communications deemed important by AAPOR Executive Council.
Our plan is to begin small. We are cognizant of the cost that professional communication can entail, and for now, we have set very modest goals. The first step is to be prepared, and have a plan for, the 2020 election, with fast response options of communication during the campaign, and perhaps most importantly, directly thereafter.
The second element of our plan is to re-envision AAPOR’s role in journalism education. In short, we believe we need to own this space, not farm it out to any other entity. We need refreshed educational videos, and many more of them, from explaining response rates to the common criticisms made on the use of horserace polling in the media.
We need to travel. Willing AAPOR members should be funded to travel and present at journalism conferences, to news rooms, and to journalism schools on an annual basis. AAPOR could as well have other live events, for example a forum on the use of polls in journalism. There should be a consistent applied effort over time. The media and journalists are AAPOR’s greatest spokespeople. By and large, much of our image is shaped through them.
The third element looks at the long game. And that is, for AAPOR to help in developing civics education on public opinion and the role of public opinion in democracy. With the help of educational experts, and importantly, tipping our hats to our AAPOR’s Got Talent winner last year, Allyson Holbrook, who proposed exactly this kind of strategy, we believe AAPOR can help develop a curriculum and educational materials and engage with educators to push for the inclusion of this curriculum in primary education. AAPOR can and should develop specific instructional objectives of civics education by grade and develop a communications plan to lobby for the inclusion of this civics curriculum by educators.
The fourth element is for AAPOR to direct the Transparency Initiative to develop a strategic plan for the next ten years. We recognize that it is not always the case that polls are executed with best practices. How does AAPOR respond in these instances? With a plethora of new sampling approaches and modalities, we believe the TI needs to have a full-throated conversation about these challenges and how AAPOR should handle them. After all, this too is part of the conversation of AAPOR communication.
Finally, AAPOR should, as past President Tim Johnson called for last year, learn as much as it can about the perceptions of polls in society. We cannot make effective strategic communication plans without knowing first how they will resonate and know to some degree their expected effectiveness. Such an effort should continue over time, building both a breadth and depth of understanding.
If this sounds a bit like a wish list, well, you would be right. For now, the immediate goal for AAPOR and its communication firm is to prepare for 2020 and to take some modest steps toward professionalizing AAPOR’s ability to effectively and quickly communicate and advocate. Looking toward the future, AAPOR Council has authorized the development of the Ad Hoc Committee on Public Opinion. This committee will be comprised of AAPOR members dedicated to pushing forward this agenda.
We recognize the potential cost of these endeavors in terms of money and labor, and so in each area, there will be mission leaders on the committee whose goal is to push forward with two goals. The first is funding. We cannot and should not fund these endeavors alone. We will be seeking foundational funding for each of these areas, and are developing a proposal for each specifically. Perhaps only one area attains funding, perhaps all of them. No matter, the committee will adjust its goals contingent on the means it has available.
A number of members have already asked to be part of these efforts. But I call on all of you, the AAPOR membership, to reach out and join the effort as well. We need people experienced in seeking funding, and people passionate in moving the needle with regard to polling journalism, civics education, and the role of public opinion in democracy. AAPOR’s secret sauce has always been the passion of its members and we call on you to help. Please go the link below to tell us you want to join the effort.
Friends and colleagues, one of the many excellent AAPOR task forces already, in fact explored this issue, the task force on polling and democracy and leadership. They argued that “AAPOR should adopt an increased public presence arguing for the importance of public opinion in a democracy and the importance of rigorous, unbiased, scientific research assessing public opinion.”
It is time we strive to realize these aspirations. For the good of our association, our field, and our very democracy. If past efforts by AAPOR volunteers are any indication, we anticipate great success and health in the future of our field and our endeavors.
It has been an honor and a privilege serving as your President. Thank you.”