32 - Making Effects Manifest in Randomized Experiments

Summary

Experimentalists want precise estimates of treatment effects and nearly always care about how treatment effects may differ across subgroups. After data collection, concern may turn to random imbalance between treatment groups on substantively important variables. Pursuit of these three goals – enhanced precision, understanding treatment effect heterogeneity, and imbalance adjustment – requires background information about experimental units. For example, one may group similar observations on the basis of such variables and then assign treatment within those blocks. Use of covariates after data have been collected raises extra concerns and requires special justification. For example, standard regression tables only approximate the statistical inference that experimentalists desire. The standard linear model may also mislead via extrapolation. After providing some general background about how covariates may, in principle, enable pursuit of precision and statistical adjustment, this chapter presents two alternative approaches to covariance adjustment: one using modern matching techniques and another using the linear model – both use randomization as the basis for statistical inference.

What Is a Manifest Effect?

A manifest effect is one we can distinguish from zero. Of course, we cannot talk formally about the effects of an experimental treatment as manifest without referring to probability: a scientist asks, “Could this result have occurred merely through chance?” or “If the true effect were zero, what is the chance that we'd observe an effect as large as this?”