Experimental Methodology

Short description:

Political science experiments are often hailed for their simplicity. But problems in the execution of experiments are common, and even in the absence of problems, design and analysis of experiments are often more complicated than textbook examples suggest. This workshop focuses on how to design and analyze experiments in light of common problems and challenges to inference. Particular topics include inattentive subjects, failures to treat subjects and other forms of noncompliance, the challenge of analyzing individual-level outcomes when randomization has been done at the group level, and the use of experiments to study causal mechanisms. We will also have some discussion of the state of the field and the directions in which it is likely to head. Examples will be drawn from areas throughout political science, with a particular focus on political psychology.

Political Psychology and Experimental Methodology: Outline

John Bullock

2012 June 30

Tuesday, sep 18, MORNING SESSION

The logic of experiments. To ensure that all students share a set of premises, I begin with a set of basic ideas about how experiments work and why they should be used. As I describe these ideas, I will introduce a system of notation and a vocabulary that I will use throughout the workshop. Even if students are familiar with the ideas, the notation and vocabulary that I use may differ from that which they have used in the past.

Tuesday, sep 18, AFTERNOON SESSION

The present and future of political science experiments. Disciplinary controversies and criticisms of experiments. The first part of this session is a broad discussion of the state of experiments in political science and the way in which experimentation is likely to change in the next decade. The main themes are the rise of field experiments and the potential decline of survey experiments.

The second and longer part of this session is a discussion of controversies about experiments and criticisms of experiments in political science. Criticisms that we will take up include the unrepresentativeness of subjects, publication bias, absence of replication efforts, difficulties associated with replication, a focus on short-lived effects, an absence of attention to behavioral outcomes, and the notion that experiments cannot be used to study the most important political problems.

Wednesday, sep 19, MORNING SESSION

Mediation analysis. Mediation analysis is the effort to understand the mechanisms through which some variables affect others. It is ubiquitous in social psychology and increasingly common in political science. But social scientists typically draw inferences about mediation without manipulating mediators, and their analyses are likely to be biased. Recognizing the problem, social scientists are gradually turning to methods that involve experimental manipulation of mediators. This is a step in the right direction, but experiments have little-appreciated limitations of their own. I will describe these limitations and argue that inference about mediation is fundamentally difficult—more difficult than inference about treatment effects. I will conclude with advice for the study of mechanisms in light of the challenges that I’ve described.

Thirsday, sep 20, AFTERNOON SESSION

Graduate student presentations.

Thirsday, sep 20, MORNING SESSION

Improving political science experiments. The quality of political science experiments has never been higher. Even so, much can be done to improve on the status quo. Some of these improvements can be made by individual experimenters; others must be implemented by journals and other professional institutions. Some are methodological (cluster your standard errors; use blocking when appropriate), while others require a more general shift in focus (think about the equilibrium effects of interventions; consider combining experiments with structural estimation).

DAY 3, AFTERNOON SESSION

Graduate student presentations.

Costs: Students admitted to the course have to provide transportation and accommodation but there are no further costs associated.

Admission: The workshop offers the opportunity for graduate students to present a working paper or a project description. Participants should email their paper before August 31 to Lasse Laustsen (ll@ps.au.dk) or Bert Bakker (bakker@sam.sdu.dk). Professor Bullock will read the paper or research design and comment on it. As it is an interactive workshop, students are encouraged to submit this document in order to gain the most from the workshop.

Accommodation

Participants are responsible for accommodation during the course themselves.