10 - How relevant is the intervention?

Summary

A third dimension along which natural experiments can be classified is the substantive relevance of the intervention. In any natural experiment, researchers and readers must ask a fundamental question of interpretation: to what extent do the effects of a random or as-if random intervention in fact shed light on wider social-scientific, substantive, and/or policy concerns?

Answers to this question might be cause for unease, for a number of reasons. For instance, the type of subjects or units exposed to the intervention might be more or less like the populations in which we are most interested. In lottery studies of electoral behavior, levels of lottery winnings may be randomly assigned among lottery players, allowing us to assess the impact of lottery winnings on political attitudes. Yet, we might doubt whether lottery players are like other populations (say, all voters). Next, the particular treatment might have idiosyncratic effects that are distinct from the effects of greatest interest. To continue the same example, levels of lottery winnings may or may not have similar effects on, say, political attitudes as income earned through work (Dunning 2008a, 2008b). Finally, natural-experimental interventions (like the interventions in some true experiments) may “bundle” many distinct treatments or components of treatments. This compounding of treatments may limit the extent to which any given natural experiment isolates the effect of an explanatory variable of greatest interest for particular substantive or social-scientific purposes. Such ideas are often discussed under the rubric of “external validity” (Campbell and Stanley 1966). Yet, the issue of substantive relevance involves a broader question: whether the intervention—based on as-if random assignment deriving from social and political processes—in fact yields causal inferences about the real causal hypothesis of concern, and for the units we would really like to study.