Hostname: page-component-586b7cd67f-tf8b9 Total loading time: 0 Render date: 2024-11-24T01:44:33.869Z Has data issue: false hasContentIssue false
  • English
  • Français

Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration

Published online by Cambridge University Press:  04 January 2017

Macartan Humphreys ,
Raul Sanchez de la Sierra  and
Peter van der Windt
Macartan Humphreys*
Affiliation:
Columbia University, Department of Political Science, 7th floor, IAB Building, 420 West 118th St., New York, NY 10027
Raul Sanchez de la Sierra
Affiliation:
Columbia University, Department of Economics, 1022 IAB Building, 420 West 118th St., New York, NY 10027 e-mail: [email protected]
Peter van der Windt
Affiliation:
Columbia University, Department of Political Science, 7th floor, IAB Building, 420 West 118th St., New York, NY 10027 e-mail: [email protected]
*
e-mail: [email protected] (corresponding author)
Get access Rights & Permissions [Opens in a new window]

Abstract

Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this “fishing” problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete “mock report” of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.

Type
Symposium on Research Registration
Information
Political Analysis , Volume 21 , Issue 1 , Winter 2013 , pp. 1 - 20
Copyright
Copyright © The Author 2013. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Authors' note: Our thanks to Ali Cirone, Andy Gelman, Grant Gordon, Alan Jacobs, Ryan Moore, and Ferran Elias Moreno for helpful comments. Our thanks to the Population Center at Columbia for providing access to the High Performance Computing (HPC) Cluster. This research was undertaken in the context of a field experiment in DRC; we thank the International Rescue Committee and CARE International for their partnership in that research and the International Initiative for Impact Evaluation (3IE) for their support. M. H. thanks the Trudeau Foundation for support while this work was undertaken. Replication data and code for tables and figures can be found at http://hdl.handle.net/1902.1/18182. Supplementary materials for this article are available on the Political Analysis Web site.

References

Beath, A., Christia, F., and Enikolopov, R. 2011. Elite capture of local institutions: Evidence from a field experiment in Afghanistan. Working paper.CrossRefGoogle Scholar
Benjamini, Y., and Hochberg, Y. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological) 57: 289–300.CrossRefGoogle Scholar
Callaway, E. 2011. Report finds massive fraud at Dutch universities. Nature 479: 15.CrossRefGoogle ScholarPubMed
Casey, K., Glennerster, R., and Miguel, E. 2011. Reshaping institutions: Evidence on external aid and local collective action. Working paper.Google Scholar
Collier, D., Seawright, J., and Munck, G. L. 2010. The quest for standards: King, Keohane, and Verba's designing social inquiry. In Rethinking social inquiry: Diverse tools, shared standards, 2nd ed., eds. Collier, D. and Brady, H. E., 33–65. London: Rowman & Littlefield Publishers.Google Scholar
De Angelis, C. D., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P., Schroeder, T. V., Sox, H. C., and Van Der Weyden, M. B. 2004. Clinical trial registration: A statement from the International Committee of Medical Journal Editors. New England Journal of Medicine 351: 1250–1.CrossRefGoogle ScholarPubMed
De Angelis, C. D., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marušic, A., Overbeke, A. J. P., Schroeder, T. V., Sox, H. C., and Weyden, M. B. V. D. 2005. Is this clinical trial fully registered? A statement from the International Committee of Medical Journal Editors. Canadian Medical Association Journal 172: 1700–2.CrossRefGoogle ScholarPubMed
Duflo, E. 2008. Using randomization in development economics research: A toolkit. In Handbook of Development Economics, eds. Strauss, J. A. and Schultz, T. P. 3895–962. North Holland: Elsevier.Google Scholar
Fearon, J. D., Humphreys, M., and Weinstein, J. M. 2009. Can development aid contribute to social cohesion after Civil War? Evidence from a field experiment in post-conflict Liberia. American Economic Review: Papers & Proceedings 99: 287–91.CrossRefGoogle Scholar
Finkelstein, A., Taubman, S., Allen, H., Gruber, J., Newhouse, J. P., Wright, B., and Baicker, K. 2010. The short-run impact of extending public health insurance to low-income adults: Evidence from the first year of the Oregon Medicaid Experiment. Working paper.CrossRefGoogle Scholar
Gelman, A., Hill, J., and Yajima, M. 2009. Why we (usually) dont have to worry about multiple comparisons. Working paper.Google Scholar
Gelman, A., and Tuerlinckx, F. 2000. Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics 15: 373–90.CrossRefGoogle Scholar
Gerber, A. S., Green, D. P., and Nickerson, D. 2001. Testing for publication bias in political science. Political Analysis 9: 385–92.CrossRefGoogle Scholar
Gerber, A., and Malhotra, N. 2008. Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science 3: 313–26.CrossRefGoogle Scholar
Glaeser, E. L. 2006. Researcher incentives and empirical methods. Working paper (DP2122).CrossRefGoogle Scholar
Hansen, P. R. 2005. A test for superior predictive ability. Journal of Business & Economic Statistics2 23: 365–80.Google Scholar
Ioannidis, J. P. A. 2005. Why most published research findings are false. PLoS Medicine 2: 696–701.CrossRefGoogle ScholarPubMed
King, G., Gakidou, E., Imai, K., Lakin, J., Moore, R. T., Nall, C., Ravishankar, N., Vargas, M., Téllez-Rojo, M. M., Avila, J. E. H., Avila, M. H., and Llamas, H. H. 2009. Public policy for the poor? A randomized assessment of the Mexican Universal Health Insurance Program. Lancet 373: 1447–54.CrossRefGoogle Scholar
King, G., Keohane, R. O., and Verba, S. 1994. Designing social inquiry: Scientific inference in qualitative research. Princeton, NJ: Princeton University Press.CrossRefGoogle Scholar
Kling, J. R., Liebman, J. B., and Katz, L. F. 2007. Experimental analysis of neighborhood effects. Econometrics 75: 83–119.CrossRefGoogle Scholar
Laine, C., Horton, R., Deangelis, C. D., Drazen, J. M., Frizelle, F. A., Godlee, F., Haug, C., Hébert, P. C., and Kotzin, S. 2007. Clinical trial registration: Looking back and moving ahead. Canadian Medical Association Journal 177: 7–8.CrossRefGoogle ScholarPubMed
Lancet. 2010. Should protocols for observational research be registered? Lancet 375: 348.CrossRefGoogle Scholar
Mathieu, S., Boutron, I., Moher, D., Altman, D. G., and Ravaud, P. 2009. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 302: 977–84.CrossRefGoogle ScholarPubMed
Monogan, J. E. III. 2010. The immigration issue and the 2010 House elections: A research design. Working paper.Google Scholar
Mutz, D., and Pemantle, R. 2011. The perils of randomization checks in the analysis of experiments. Working paper.Google Scholar
Rasmussen, O. D., Malchow-Møller, N., and Barnebeck Andersen, T. 2011. Walking the talk: The need for a trial registry for development interventions. Journal of Development Effectiveness 3: 1–29.Google Scholar
Reveiz, L., Chan, A.-W., Krleza-Jerić, K., Granados, C. E., Pinart, M., Etxeandia, I., Rada, D., Martinez, M., Bonfill, X., and Cardona, A. F. 2010. Reporting of methodologic information on trial registries for quality assessment: A study of trial records retrieved from the WHO search portal. PloS One 5: 1–6.CrossRefGoogle ScholarPubMed
Simmons, J. P., Nelson, L. D., and Simonsohn, U. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22: 1359–66.CrossRefGoogle ScholarPubMed
Sterling, T. D. 1959. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association 54: 30–34.Google Scholar
White, H. 2000. A reality check for data snooping. Econometrica 68: 1097–126.CrossRefGoogle Scholar
Supplementary material: File

Humphreys et al. supplementary material

Appendix A

Download Humphreys et al. supplementary material(File)
File 9.7 KB
Supplementary material: File

Humphreys et al. supplementary material

Appendix B

Download Humphreys et al. supplementary material(File)
File 15 KB
Supplementary material: File

Humphreys et al. supplementary material

Appendix C

Download Humphreys et al. supplementary material(File)
File 10.6 KB