D01-01AM: Designing Survey Experiments

Survey-embedded experiments have become a major method to make credible causal inferences about citizens’ and groups’ attitudes, preferences and behavior across social science disciplines. Findings of survey experiments promise to be more generalizable than those of many field experiments and implementation is usually cheaper than in lab experiments. The course provides an advanced introduction to the logic, design, popular types, challenges and pitfalls of survey experiments.

We will first discuss basic concepts of experiments and their relevance in the survey context. On this basis, we survey popular types of survey experiments to test the impact of information or frames on attitudes (e.g. framing experiments), elicit complex preferences (conjoint/vignette experiments), and uncover sensitive information and opinions (e.g. list experiments) from respondents. Finally, we consider common challenges and pitfalls that are particularly pervasive in survey experiments such as respondents inferring the purpose of the experiment or more information than is provided, lack of manipulation, erroneous interpretations of treatments and effects, or lack of ecological validity. The course will not cover the statistical analysis of survey experiments, since it is straightforward in many cases. Participants will have the chance to receive feedback on their own designs or design ideas.

Course Schedule

The sessions will start with an interactive lecture element. In the second half, participants will have the opportunity to present some of their survey-experimental design ideas that will be discussed in class. Depending on how many participants want to present what kinds of experiments, the schedule will be adjusted.

Day 1 – Foundations of (Survey) Experiments

Why are experiments and particularly survey experiments so popular in the social sciences? What is the basic logic of experiments? What are the basic concepts we need to know? How can experiments achieve causal inference?

Concepts covered: Selection bias, observables/unobservables, randomization, covariate balance, treatment, manipulation, compliance, treatment effect, intention-to-treat effect, stable unit treatment value assumption, spillover

Indicative readings:
Chapter 2 in Angrist, Joshua D., and Jörn-Steffen Pischke. 2008. Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton: Princeton University Press.
Holland, Paul W. 1986. “Statistics and Causal Inference.” Journal of the American Statistical Association 81(396): 945–60.
Chapter 1 in Mutz, Diana C. 2011. Population-Based Survey Experiments. Princeton University Press.

Day 2 – Survey Experiments with Single Treatments

How can we use survey experiments to identify the effect of a single causal factor, such as a particular piece of information or a particular frame in which something is presented, on people’s attitudes and opinions? What are different types of treatments in survey experiments?

Concepts covered: Direct and indirect treatments, priming experiments, framing experiments, manipulation checks

Indicative readings:
Chapter 3 in Mutz, Diana C. 2011. Population-Based Survey Experiments. Princeton University Press.
Tomz, Michael R., and Jessica LP Weeks. 2013. “Public Opinion and the Democratic Peace.” American Political Science Review 107(4): 849-865.
Busby, Ethan C, Joshua R Gubler, and Kirk A Hawkins. 2019. “Framing and Blame Attribution in Populist Rhetoric.” Journal of Politics 81(2): 616-630.

Day 3 – Survey Experiments with High-Dimensional Treatments

How can we use survey experiments to understand complex preferences of citizens and isolate factors that are more/less important for the formation of these preferences? What are different forms of “stated preference experiments”, and which one should we choose? What are identifying assumptions of such designs?

Concepts covered: Conjoint vs. vignette, choice task, single vs. paired profiles, attributes and attribute levels, marginal means, average marginal component effects, no carryover effects assumption, no profile order effects assumption

Indicative readings:
Hainmueller, Jens, Daniel J. Hopkins, and Teppei Yamamoto. 2014. “Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments.” Political Analysis 22(1): 1–30.
Hainmueller, Jens, Dominik Hangartner, and Teppei Yamamoto. 2015. “Validating Vignette and Conjoint Survey Experiments against Real-World Behavior.” Proceedings of the National Academy of Sciences of the United States of America 112(8): 2395–2400.
Hainmueller, Jens, and Daniel J. Hopkins. 2015. “The Hidden American Immigration Consensus: A Conjoint Analysis of Attitudes toward Immigrants.” American Journal of Political Science 59(3): 529–48.

Day 4 – Survey Experiments to Elicit Sensitive Information

How can we use survey experiments to obtain truthful information from respondents about sensitive topics such as socially (un)desirable opinions or behaviors (e.g. having voted in elections, criminal offences, racist attitudes)? What are different types of experiments to reduce social desirability bias?

Concepts covered: List experiment, endorsement experiment, no design effects assumption, ceiling/floor effects, difference-in-means estimator

Indicative readings:
Blair, Graeme, Kosuke Imai, and Jason Lyall. 2014. “Comparing and Combining List and Endorsement Experiments: Evidence from Afghanistan.” American Journal of Political Science 58(4): 1043–63.
Glynn, Adam N. 2013. “What Can We Learn with Statistical Truth Serum? Design and Analysis of the List Experiment.” Public Opinion Quarterly 77(S1): 159–72.
Gonzalez-Ocantos, Ezequiel et al. 2012. “Vote Buying and Social Desirability Bias: Experimental Evidence from Nicaragua.” American Journal of Political Science 56(1): 202–17.

Day 5 – Common Challenges and Pitfalls of Survey Experiments

What are common challenges and pitfalls that researchers face when designing and analyzing survey experiments? How much should we worry about them? How can they be addressed?

Concepts covered: Demand effects, information equivalence, external/ecological validity, idiosyncrasies of treatment implementation, misinterpretations of AMCE

Indicative readings:
Mummolo, Jonathan, and Erik Peterson. 2019. “Demand Effects in Survey Experiments: An Empirical Assessment.” American Political Science Review 113(2): 517-529.
Brutger, Ryan, Joshua D. Kertzer, Jonathan Renshon, Dustin Tingley, and Chagai M. Weiss. 2023. “Abstraction and Detail in Experimental Design.” American Journal of Political Science 67(4): 979-995.
Dafoe, Allan, Baobao Zhang, and Devin Caughey. 2018. “Information Equivalence in Survey Experiments.” Political Analysis 26(4): 399-416.

This description is subject to change at the discretion of the Instructor

2 Credits

For completion of all work before and during the course, as outlined by the Instructor, and 90% participation and attendance of the course.

2 Additional Credits

Course specific extra assignments. These can include submitting assignments before the course, daily assignments, and/or a final assignment to be completed after the course as decided by the Instructor.