JPSM Short Courses-Experimental Design for Surveys

A key tool of methodological research is the split-ballot experiment, in which randomly selected subgroups of a sample receive different questions, different response formats, or different modes of data collection.  In theory, such experiments can combine the clarity of experimental designs with the inferential power of representative samples. All too often, though, such experiments use flawed designs that leave serious doubts about the meaning or generalizability of the findings.  The purpose of this course is to consider the issues involved in the design and analysis of data from experiments embedded in surveys.  It covers the purposes of experiments in surveys, examines several classic survey experiments in detail, and takes a close look at some of the pitfalls and issues in the design of such studies.  These pitfalls include problems (such as the confounding of the experimental variables) that jeopardize the comparability of the experimental groups, problems (such as nonresponse) that cast doubts on the generality of the results, and problems in determining the reliability of the results. The course will also consider some of the design decisions that almost always arise in planning experiments — issues such as identifying the appropriate error term for significance tests and including necessary comparison groups.  For information, https://projects.isr.umich.edu/jpsm/