5 - Simplicity and transparency

Summary

Suppose that a researcher has discovered a credible natural experiment and designed a study akin to the best examples I have surveyed in previous chapters. This researcher then faces a central question: how should the data be analyzed? This second part of the book takes up this central question.

For reasons I will discuss extensively, the analysis of natural experiments should ideally leverage both qualitative and quantitative methods. There are important choices to be made with respect to both modes of analysis. This chapter and the next discuss quantitative analysis, leaving qualitative tools to Chapter 7. In this chapter, I describe the definition and estimation of causal effects. The role of chance variability—including assumptions about sampling processes and procedures for calculating standard errors—is left for Chapter 6.

A central theme of this chapter and the next is that the quantitative analysis of natural experiments can and should be very simple. Often, a minimum of mathematical manipulation is involved. For example, straightforward contrasts between the treatment and control groups—such as the difference in average outcomes in these two groups—often suffices to provide evidence of causal effects. Such difference-of-means (or difference-of-percentages) analysis plays a central role in each kind of natural experiment, from standard natural experiments and regression discontinuities to instrumental-variables designs.

The potential simplicity of quantitative data analysis provides an important advantage of natural experiments, for a number of reasons. First, it implies that statistical results are often easy to convey and interpret. Generally, just one or a few main quantitative tests are most relevant to testing key hypotheses. Rather than presenting the estimated coefficients from multivariate models in long tables of regression results, analysts may have more space in articles to discuss the research design and substantive import of the results.