Hostname: page-component-cd9895bd7-q99xh Total loading time: 0 Render date: 2024-12-27T07:05:38.170Z Has data issue: false hasContentIssue false
  • English
  • Français

The sensitivity of sensitivity analysis

Published online by Cambridge University Press:  29 August 2018

Thomas Plümper  and
Richard Traunmüller
Thomas Plümper*
Affiliation:
Department of Socioeconomics, Vienna University of Economics, Vienna, Austria
Richard Traunmüller
Affiliation:
Department of Social Sciences, Goethe University Frankfurt am Main, Germany
*
*Corresponding author. Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

This article evaluates the reliability of sensitivity tests. Using Monte Carlo methods we show that, first, the definition of robustness exerts a large influence on the robustness of variables. Second and more importantly, our results also demonstrate that inferences based on sensitivity tests are most likely to be valid if determinants and confounders are almost uncorrelated and if the variables included in the true model exert a strong influence on outcomes. Third, no definition of robustness reliably avoids both false positives and false negatives. We find that for a wide variety of data-generating processes, rarely used definitions of robustness perform better than the frequently used model averaging rule suggested by Sala-i-Martin. Fourth, our results also suggest that Leamer’s extreme bounds analysis and Bayesian model averaging are extremely unlikely to generate false positives. Thus, if based on these inferential criteria a variable is robust, it is almost certain to belong into the empirical model. Fifth and finally, we also show that researchers should avoid drawing inferences based on lack of robustness.

Type
Research Notes
Information
Political Science Research and Methods , Volume 8 , Issue 1 , January 2020 , pp. 149 - 159
Copyright
Copyright © The European Political Science Association 2018 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bartels, LM (1997) Specification Uncertainty and Model Averaging. American Journal of Political Science 41, 641–674.Google Scholar
Cederman, LE, Wimmer, A Min, B (2010) Why do Ethnic Groups Rebel. New Data and Analysis. World Politics 62(1), 87–119.Google Scholar
Gleditsch, KS (2007) Transnational Dimensions of Civil War. Journal of Peace Research 44(3), 293–309.Google Scholar
Granger, CWJ Uhlig, HF (1990) Reasonable Extreme Bounds Analysis. Journal of Econometrics 44, 159–170.Google Scholar
Hafner-Burton, EM (2005) Right or Robust. The Sensitive Nature of Repression to Globalization. Journal of Peace Research 42(6), 679–698.Google Scholar
Hegre, H Sambanis, N (2006) Sensitivity Analysis of Empirical Results on Civil War Onset. Journal of Conflict Resolution 50(4), 508–535.Google Scholar
Hoeting, JA, Madigan, D, Raftery, AE Volinsky, CT (1999) Bayesian Model Averaging: A Tutorial. Statistical Science 14, 382–401.Google Scholar
Leamer, EE (1978) Specification Searches: Ad Hoc Inference with Nonexperimental Data. New York: Wiley.Google Scholar
Leamer, EE (1983) Let’s Take the Con Out of Econometrics. American Economic Review 73(1), 31–43.Google Scholar
Leamer, EE (1985) Sensitivity Analysis Would Help. American Economic Review 57(3), 308–313.Google Scholar
Leamer, EE (2010) Tantalus on the Road to Asymptopia. The Journal of Economic Perspectives 24(2), 31–46.Google Scholar
Leamer, E Leonard, H (1983) Reporting the Fragility of Regression Estimates. The Review of Economics and Statistics 65(2), 306–317.Google Scholar
Levine, R Renelt, D (1992) A Sensitivity Analysis of Cross-Country Growth Regressions. American Economic Review 82(4), 942–963.Google Scholar
Neumayer, E Plümper, T (2017) Robustness Tests for Quantitative Research. Cambridge: Cambridge University Press.Google Scholar
Raftery, AE (1995) Bayesian Model Selection in Social Research. Sociological Methodology 25, 111–163.Google Scholar
Sala-i-Martin, X (1997) I Just Ran Two Million Regressions. American Economic Review 87(2), 178–183.Google Scholar
Sturm, JE, Berger, H de Haan, J (2005) Which Variables Explain Decisions on IMF Credit. An Extreme Bounds Analysis. Economics & Politics 17(2), 177–213.Google Scholar
Sturm, JE de Haan, J (2005) Determinants of Long-Term Growth: New Results Applying Estimation and Extreme Bounds Analysis. Empirical Economics 30(3), 597–617.Google Scholar
Supplementary material: Link

Plümper and Traunmüller Dataset

Dataset

https://doi.org/10.7910/DVN/PBEOJZ
Link