Hostname: page-component-586b7cd67f-2brh9 Total loading time: 0 Render date: 2024-11-30T21:54:19.240Z Has data issue: false hasContentIssue false
  • English
  • Français

A Comment on Diagnostic Tools for Counterfactual Inference

Published online by Cambridge University Press:  04 January 2017

Nicholas Sambanis  and
Alexander Michaelides
Nicholas Sambanis*
Affiliation:
Department of Political Science, Yale University, PO Box 208301, New Haven, CT 06520
Alexander Michaelides
Affiliation:
London School of Economics, Department of Economics, Houghton Street, London WC2A 2AE, UK, e-mail: [email protected]
*
e-mail: [email protected] (corresponding author)
Get access Rights & Permissions [Opens in a new window]

Abstract

We evaluate two diagnostic tools used to determine if counterfactual analysis requires extrapolation. Counterfactuals based on extrapolation are model dependent and might not support empirically valid inferences. The diagnostics help researchers identify those counterfactual “what if” questions that are empirically plausible. We show, through simple Monte Carlo experiments, that these diagnostics will often detect extrapolation, suggesting that there is a risk of biased counterfactual inference when there is no such risk of extrapolation bias in the data. This is because the diagnostics are affected by what we call the n/k problem: as the number of data points relative to the number of explanatory variables decreases, the diagnostics are more likely to detect the risk of extrapolation bias even when such risk does not exist. We conclude that the diagnostics provide too severe a test for many data sets used in political science.

Type
Research Article
Information
Political Analysis , Volume 17 , Issue 1 , Winter 2009 , pp. 89 - 106
Copyright
Copyright © The Author 2008. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Author's note: We thank Komei Fukuda, Don Green, Alan Gerber, and Jasjeet Sekhon for their generous help, Mike Kane for assistance with R programming, and five anonymous referees for constructive comments.

References

Angrist, Joshua D., and Krueger, Alan B. 2001. Instrumental variables and the search for identification: from supply and demand to natural experiments. Working paper #455. Princeton University Industrial Relations Section.CrossRefGoogle Scholar
Becker, Sasha O., and Ichino, Andrea. 2002. Estimation of average treatment effects based on propensity scores. The Stata Journal 2: 358–77.CrossRefGoogle Scholar
Elekes, Gyorgi. 1986. A geometric inequality and the complexity of computing volume. Discrete and Computational Geometry 1: 289–92.CrossRefGoogle Scholar
Gower, J. C. 1966. Some distance properties of latent root and vector methods used in multivariate analysis. Biometrika 53: 325–88.CrossRefGoogle Scholar
Gower, J. C. 1971. A general coefficient of similarity and some of its properties. Biometrics 27: 857–72.CrossRefGoogle Scholar
Ho, Daniel E., Imai, Kosuke, King, Gary, and Stuart, Elizabeth A. 2007. Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis 15: 199–236.CrossRefGoogle Scholar
King, Gary, and Zeng, Langche. 2006. The dangers of extreme counterfactuals. Political Analysis 14: 131–59.CrossRefGoogle Scholar
Montalvo, Jose G., and Reynal-Querol, Marta. 2005. Ethnic polarization, potential conflict, and civil wars. American Economic Review 95: 796–816.CrossRefGoogle Scholar
Sekhon, Jasjeet S. 2007. Multivariate and propensity score matching software with automated balance optimization: The matching package for R. http://sekhon.berkeley.edu/matching.Google Scholar