Expert Insights on Economic Survey Research with Diane Willimack


Krysten Mesner, U.S. Census Bureau

Recently, I had the privilege of sitting down with Diane Willimack – Senior Survey Methodologist for Economic Programs at the U.S. Census Bureau – to gather some of her insights on economic survey research.  Diane is an expert in the field of economic survey methodology, pioneering, in particular, the cognitive aspects of response to economic surveys. Her work has a distinctively mixed methods approach: blending quantitative analyses on paradata and response data with qualitative collection from business respondents across the economy.

We spent the afternoon discussing exciting topics such as administrative data, instrument design, response rates, incentives in business surveys, machine learning, paradata, and more.  Below is a condensed summary of our time together.

KM:  How did you end up in this field?

DW:  I found that I was more interested in where the data came from instead of what the data were.  The data are not exact and that’s what got me interested in where the data come from: your measurement is always with error.  You’re never going to have a perfect number. Having been an economist, I wanted  my peers (economists) to understand that the data are not the full truth, and that you’re never going to get an exact number. The qualitative work is meant to understand and describe the discrepancy between what the economist wants and they data they get.  Economists should be attentive to the qualitative results because they have the knowledge to understand the impact of the difference between the reported data and the concept from their theory.

KM:  What’s on the horizon for economic survey research?

The use of third-party and administrative data.  The statisticians have a role in determining how good those data are and the qualitative researchers would be looking at what those data consist of.  And, the statisticians will — or should — be taking a stab at understanding and quantifying that discrepancy because the administrative data is not the same as the respondent data.

It seems to me that the field thinks that administrative records are the truth, but obtaining those administrative data required asking questions, and often times filling out a form.  That sounds to me like questionnaire design and cognitive research, and I think that’s been overlooked in terms of administrative records. It still starts with a form and a person interpreting questions.

As for system-to-system response – where computer systems from companies connect directly to computer systems from the government for reporting data — there’s still a human involved.  By the businesses giving us data straight from their records, we are missing the part of ‘what the data means,’ which is concerning. If we are getting data directly from the businesses without an intervention then we are missing what the data means.

When I was talking to directly to businesses and [trying to figure out] what they could provide in terms of system-to-system, the common reaction from business respondents was – the researchers are going to need to talk to businesses to let businesses help the researchers interpret it. I do not discount these developments and advancements that will be made because of technology but as long as there are people involved then it’s not straightforward. It’s something that needs to be in the pipeline with all of the other research that we are doing.

KM:  What guidance would you give for economic survey researchers?

Be astute and flexible. Always be collaborative.  Be respectful of your peers and your subjects – your respondents.  Understand where they’re coming from.  Ask questions.

When I started, just about anything we did was new and could have been publishable as it had never been done before. Now I don’t think you can say that.  The methods we developed and used originally may have come from surveys of households and individuals, but using them for business surveys was unexpectedly not straightforward. There are still all kinds of never been done before things – but now you have practices, frameworks, and methodologies that are ours to use to understand.  And that’s what you can make your career out of.

Establishment surveys are so interesting. Establishment surveys have a lot of different elements to investigate for researchers like us. A lot. I made a 40-year career out of it.