D10: Designing Survey Experiments

Survey-embedded experiments have become a major method to make credible causal inferences about citizens’ and groups’ attitudes, preferences and behavior across social science disciplines. Findings of survey experiments promise to be more generalizable than those of many field experiments and implementation is usually cheaper than in lab experiments. The course provides an advanced introduction to the logic, design, popular types, challenges and pitfalls of survey experiments. We will first discuss basic concepts of experiments and their relevance in the survey context. On this basis, we survey popular types of survey experiments to test the impact of information or frames on attitudes (e.g. framing experiments), elicit complex preferences (conjoint/vignette experiments), and uncover sensitive information and opinions (e.g. list experiments) from respondents. Finally, we consider common challenges and pitfalls that are particularly pervasive in survey experiments such as respondents inferring the purpose of the experiment or more information than is provided, lack of manipulation, erroneous interpretations of treatments and effects, or lack of ecological validity. The course will not cover the statistical analysis of survey experiments, since it is straightforward in many cases. Participants will have the chance to receive feedback on their own designs or design ideas.

Survey-embedded experiments have become a major method to make credible causal inferences about citizens’ and groups’ attitudes, preferences and behavior across social science disciplines. Findings of survey experiments promise to be more generalizable than those of many field experiments and implementation is usually cheaper than in lab experiments. The course provides an advanced introduction to the logic, design, popular types, challenges and pitfalls of survey experiments. We will first discuss basic concepts of experiments and their relevance in the survey context. On this basis, we survey popular types of survey experiments to test the impact of information or frames on attitudes (e.g. framing experiments), elicit complex preferences (conjoint/vignette experiments), and uncover sensitive information and opinions (e.g. list experiments) from respondents. Finally, we consider common challenges and pitfalls that are particularly pervasive in survey experiments such as respondents inferring the purpose of the experiment or more information than is provided, lack of manipulation, erroneous interpretations of treatments and effects, or lack of ecological validity. The course will not cover the statistical analysis of survey experiments, since it is straightforward in many cases. Participants will have the chance to receive feedback on their own designs or design ideas.

The sessions will start with an interactive lecture element that engages with the contents of the readings as well as additional material. In the second half, we will do some (group) exercises to practice core design elements of survey experiments. In addition, participants will have the opportunity to present some of their survey-experimental design ideas that will be discussed in class. Depending on how many participants want to present what kinds of experiments, the schedule will be adjusted.

Day 1 – Foundations of (Survey) Experiments

Why are experiments and particularly survey experiments so popular in the social sciences? What is the basic logic of experiments? What are the basic concepts we need to know? How can experiments achieve causal inference? 

Concepts covered: 

Selection bias, observables/unobservables, randomization, covariate balance, treatment, treatment effect, stable unit treatment value assumption, spillover

Day 2 – Survey Experiments with Single Treatments

How can we use survey experiments to identify the effect of a single causal factor, such as a particular piece of information or a particular frame in which something is presented, on people’s attitudes and opinions? What are different types of treatments in survey experiments? 

Concepts covered:

Framing experiments, manipulation, direct and indirect treatments, compliance, intention-to-treat effect, manipulation checks, complier average causal effect

Day 3 – Survey Experiments with High-Dimensional Treatments

How can we use survey experiments to understand complex preferences of citizens and isolate factors that are more/less important for the formation of these preferences? What are different forms of “stated preference experiments”, and which one should we choose? What are identifying assumptions of such designs?

Concepts covered:

Conjoint vs. vignette, choice task, single vs. paired profiles, attributes and attribute levels, average marginal component effects, no carryover effects assumption, no profile order effects assumption

Day 4 – Survey Experiments to Elicit Sensitive Information

How can we use survey experiments to obtain truthful information from respondents about sensitive topics such as socially (un)desirable opinions or behaviors (e.g. having voted in elections, criminal offences, racist attitudes)? What are different types of experiments to reduce social desirability bias? 

Concepts covered:

List experiment, endorsement experiment, randomized response experiment, no design effects assumption, no liars assumption, ceiling/floor effects, difference-in-means estimator

Day 5 – Common Challenges and Pitfalls of Survey Experiments

What are common challenges and pitfalls that researchers face when designing and analyzing survey experiments? How much should we worry about them? How can they be addressed?

Concepts covered:

Demand effects, information equivalence, external/ecological validity, idiosyncrasies of treatment implementation, misinterpretations of AMCE

Indicative Readings:

Mutz, Diana C. 2011. Population-Based Survey Experiments. Princeton University Press.

Hainmueller, Jens, Daniel J. Hopkins, and Teppei Yamamoto. 2014. “Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments.” Political Analysis 22(1): 1–30.

Hainmueller, Jens, Dominik Hangartner, and Teppei Yamamoto. 2015. “Validating Vignette and Conjoint Survey Experiments against Real-World Behavior.” Proceedings of the National Academy of Sciences of the United States of America 112(8): 2395–2400.

Blair, Graeme, Kosuke Imai, and Jason Lyall. 2014. “Comparing and Combining List and Endorsement Experiments: Evidence from Afghanistan.” American Journal of Political Science 58(4): 1043–63.

Glynn, Adam N. 2013. “What Can We Learn with Statistical Truth Serum? Design and Analysis of the List Experiment.” Public Opinion Quarterly 77(S1): 159–72.

Mummolo, Jonathan, and Erik Peterson. 2019. “Demand Effects in Survey Experiments: An Empirical Assessment.” American Political Science Review 113(2): 517-529. 

Brutger, Ryan, Joshua D. Kertzer, Jonathan Renshon, Dustin Tingley, and Chagai M. Weiss. 2023. “Abstraction and Detail in Experimental Design.” American Journal of Political Science 67(4): 979-995.

Dafoe, Allan, Baobao Zhang, and Devin Caughey. 2018. “Information Equivalence in Survey Experiments.” Political Analysis 26(4): 399-416.

This description is subject to change at the discretion of the Instructor

2 Credits

For completion of all work before and during the course, as outlined by the Instructor, and 90% participation and attendance of the course.

2 Additional Credits

Course specific extra assignments. These can include submitting assignments before the course, daily assignments, and/or a final assignment to be completed after the course as decided by the Instructor.