Article contents
The External Validity of College Student Subject Pools in Experimental Research: A Cross-Sample Comparison of Treatment Effect Heterogeneity
Published online by Cambridge University Press: 19 October 2018
Abstract
Are student subject experiment pools comparable across institutions? Despite repeated concerns over the “college sophomore problem,” many experiment-based studies still rely on student subject pools due to their convenience and accessibility. In this paper, I investigate whether student subject pools are comparable across universities by examining how respondents across three student subject pools at distinct educational institutions perform on the same survey experiment about crisis bargaining between states. I argue that, due to selection biases inherent in university matriculation and the self-selection of students into experimental protocols, respondents across these subject pools will exhibit key demographic differences. I also examine whether respondents across these subject pools think similarly about international politics and respond comparably to experimental treatments. I find that, while there are significant demographic differences across subject pools, subjects across institutions respond similarly to experimental treatments—with the key exception of information regarding the regime type of a state. Furthermore, there is little evidence that these demographic differences impact conditional average treatment effects across subgroups. These findings carry critical implications for the use of student samples across political science and within international relations more specifically, particularly regarding the current replication crisis in the discipline.
- Type
- Articles
- Information
- Copyright
- Copyright © The Author(s) 2018. Published by Cambridge University Press on behalf of the Society for Political Methodology.
Footnotes
Contributing Editor: R. Michael Alvarez
Author’s note: Thank you to Matt Luttig and Aila Mattanock for their comments on earlier drafts of this paper. Thank you also to Chris Gelpi, Tim Büthe, and Bill Boettcher for their comments on the survey instrument. This work was supported by funding from Duke University and Colgate University. This research was approved by Institutional Review Boards at Colgate University (#ER-S15-33), Duke University (#B0170), and North Carolina State University (#2999). Replication files are available at Lupton (2018b).
References
- 20
- Cited by