Hostname: page-component-55f67697df-zh294 Total loading time: 0 Render date: 2025-05-13T06:04:46.977Z Has data issue: false hasContentIssue false
  • English
  • Français

A practical guide to Registered Reports for economists

Published online by Cambridge University Press:  17 January 2025

Thibaut Arpinon  and
Romain Espinosa Open the ORCID record for Romain Espinosa [Opens in a new window]
Thibaut Arpinon
Affiliation:
CREM, University of Rennes 1, Rennes, France
Romain Espinosa*
Affiliation:
CIRED, CNRS, Nogent-sur-Marne, France
*
e-mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

The current publication system in economics has encouraged the inflation of positive results in empirical papers. Registered Reports, also called Pre-Results Reviews, are a new submission format for empirical work that takes pre-registration one step further. In Registered Reports, researchers write their papers before running the study and commit to a detailed data collection process and analysis plan. After a first-stage review, a journal can give an In-Principle-Acceptance guaranteeing that the paper will be published if the authors carry out their data collection and analysis as pre-specified. We here propose a practical guide to Registered Reports for empirical economists. We illustrate the major problems that Registered Reports address (p-hacking, HARKing, forking, and publication bias), and present practical guidelines on how to write and review Registered Reports (e.g., the data-analysis plan, power analysis, and correction for multiple-hypothesis testing), with R and STATA codes. We provide specific examples for experimental economics, and show how research design can be improved to maximize statistical power. Last, we discuss some tools that authors, editors, and referees can use to evaluate Registered Reports (checklist, study-design table, and quality assessment).

Keywords

Registered ReportsPractical guidePre-registrationp-hackingHARKingMultiple-hypothesis testingPower analysisThe smallest effect size of interest

JEL classification

A10: General C12: Hypothesis Testing: General C9: Design of Experiments
Type
Methodology Paper
Information
Journal of the Economic Science Association , Volume 9 , Issue 1 , June 2023 , pp. 90 - 122
Copyright
Copyright © The Author(s), under exclusive licence to Economic Science Association 2023.

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

The authors thank Lionel Page, Emma Henderson, Daniel Lakens, Zoltan Dienes, Jens Rommel, Anna Dreber Almenberg, Andrew Clark, Marianne Lefebvre, and Etienne Dagorn for useful comments.

References

Abrams, E., Libgober, J., & List, J. A. (2020). Research registries: Facts, myths, and possible improvements. NBER Working Paper.CrossRefGoogle Scholar
Agnoli, F., Wicherts, J. M., Veldkamp, C. L., Albiero, P., Cubelli, R. (2017). Questionable research practices among Italian research psychologists. PloS One, 12(3), 10.1371/journal.pone.0172792CrossRefGoogle ScholarPubMed
Albers, C., Lakens, D. (2018). When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias. Journal of Experimental Social Psychology, 74, 187–195. 10.1016/j.jesp.2017.09.004CrossRefGoogle Scholar
Althouse, A. D. (2021). Post hoc power: Not empowering, just misleading. Journal of Surgical Research, 259, A3–A6. 10.1016/j.jss.2019.10.049CrossRefGoogle ScholarPubMed
Altman, D. G., Bland, J. M. (1995). Statistics notes: Absence of evidence is not evidence of absence. BMJ, 311(7003),485. 10.1136/bmj.311.7003.485CrossRefGoogle Scholar
Amrhein, V., Greenland, S., McShane, B. (2019). Scientists rise up against statistical significance. Nature, 567, 305–307. 10.1038/d41586-019-00857-9CrossRefGoogle ScholarPubMed
Bakker, M., Veldkamp, C. L., van Assen, M. A., Crompvoets, E. A., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLoS Biology, 18(12), 10.1371/journal.pbio.3000937CrossRefGoogle ScholarPubMed
Beaumont, J.-F., Rivest, L.-P. (2009). Dealing with outliers in survey data Handbook of statistics, Elsevier 247–279.Google Scholar
Bellemare, C., Bissonnette, L., Kröger, S. (2016). Simulating power of economic experiments: The powerbbk package. Journal of the Economic Science Association, 2(2), 157–168. 10.1007/s40881-016-0028-4CrossRefGoogle Scholar
Brodeur, A., Cook, N., Hartley, J., & Heyes, A. (2022). Do pre-registration and pre-analysis plans reduce p-hacking and publication bias? Available at SSRN.CrossRefGoogle Scholar
Brodeur, A., Lé, M., Sangnier, M., Zylberberg, Y. (2016). Star wars: The empirics strike back. American Economic Journal: Applied Economics, 8(1), 1–32.Google Scholar
Bruns, S., Deressa, T. K., Stanley, T., Doucouliagos, C., & Ioannidis, J. (2022). Estimating the extent of inflated significance in economics. MetaArXiv—Preprints.CrossRefGoogle Scholar
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T. et al., (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436. 10.1126/science.aaf0918CrossRefGoogle ScholarPubMed
Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., Etchells, P. (2014). Instead of playing the game it is time to change the rules: Registered reports at aims neuroscience and beyond. AIMS Neuroscience, 1(1), 4–17. 10.3934/Neuroscience.2014.1.4CrossRefGoogle Scholar
Chambers, C. D., Tzavella, L. (2022). The past, present and future of registered reports. Nature Human Behaviour, 6, 29–42. 10.1038/s41562-021-01193-7CrossRefGoogle ScholarPubMed
Chen, R., Chen, Y., Riyanto, Y. E. (2021). Best practices in replication: A case study of common information in coordination games. Experimental Economics, 24(1), 2–30. 10.1007/s10683-020-09658-8CrossRefGoogle Scholar
Christensen, G., Miguel, E. (2018). Transparency, reproducibility, and the credibility of economics research. Journal of Economic Literature, 56(3), 920–80. 10.1257/jel.20171350CrossRefGoogle Scholar
Clarke, D., Romano, J. P., Wolf, M. (2020). The Romano–Wolf multiple-hypothesis correction in Stata. The Stata Journal, 20(4), 812–843. 10.1177/1536867X20976314CrossRefGoogle Scholar
Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science, 6(3), 274–290. 10.1177/1745691611406920CrossRefGoogle Scholar
Dienes, Z. (2020). The inner workings of registered reports. PsyArXiv—Preprints.CrossRefGoogle Scholar
Dienes, Z. (2021). Obtaining evidence for no effect. Collabra: Psychology, 7(1),28202. 10.1525/collabra.28202CrossRefGoogle Scholar
Dienes, Z. (2022). Testing theories with Bayes factors. PsyArXiv—Preprints.Google Scholar
Dutilh, G., Sarafoglou, A., Wagenmakers, E.-J. (2021). Flexible yet fair: Blinding analyses in experimental psychology. Synthese, 198(23), 5745–5772. 10.1007/s11229-019-02456-7CrossRefGoogle Scholar
Espinosa, R., Arpinon, T., Maginot, P., Demange, S., & Peureux, F. (2022). Removing barriers to plant-based diets: assisting doctors with vegan patients. Stage-2 Registered Report accepted at PCI RR.Google Scholar
Espinosa, R., Treich, N. (2021). Moderate versus radical NGOs. American Journal of Agricultural Economics, 103(4), 1478–1501. 10.1111/ajae.12156CrossRefGoogle Scholar
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PloS One, 4(5), 10.1371/journal.pone.0005738CrossRefGoogle ScholarPubMed
Fanelli, D. (2010). Positive results increase down the hierarchy of the sciences. PloS One, 5(4), 10.1371/journal.pone.0010068CrossRefGoogle ScholarPubMed
Ferraro, P. J., & Shukla, P. (2020). Feature—Is a replicability crisis on the horizon for environmental and resource economics? Review of Environmental Economics and Policy, 14(2).CrossRefGoogle Scholar
Fiedler, K., Schwarz, N. (2016). Questionable research practices revisited. Social Psychological and Personality Science, 7(1), 45–52. 10.1177/1948550615612150CrossRefGoogle Scholar
Franco, A., Malhotra, N., Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. 10.1126/science.1255484CrossRefGoogle ScholarPubMed
Gelman, A., Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no fishing expedition or p-hacking and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University, 348, 1–17.Google Scholar
Glennerster, R., Takavarasha, K. (2013). Running randomized evaluations Running randomized evaluations, Princeton University Press 10.2307/j.ctt4cgd52CrossRefGoogle Scholar
Heckelei, T., Hüttel, S., Odening, M., Rommel, J. (2022). The replicability crisis and the p-value debate—What are the consequences for the agricultural and food economics community? Preprints.CrossRefGoogle Scholar
Henderson, E. (2022). A guide to preregistration and registered reports. MetaArXiv.CrossRefGoogle Scholar
Henderson, E. L., Chambers, C. D. (2022). Ten simple rules for writing a registered report. PLoS Computational Biology, 18(10), 10.1371/journal.pcbi.1010571CrossRefGoogle ScholarPubMed
Henderson, E., Vallée-Tourangeau, F., & Simons, D. (2019). The effect of concrete wording on truth judgements: A preregistered replication and extension of Hansen & Wänke (2010). Registered Report—Stage 1, retrieved from osf.io/f9jh6.Google Scholar
Jasielska, D., Rogoza, R., Zajenkowska, A., Russa, M. B. (2021). General trust scale: Validation in cross-cultural settings. Current Psychology, 40(10), 5019–5029. 10.1007/s12144-019-00435-2CrossRefGoogle Scholar
John, L. K., Loewenstein, G., Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. 10.1177/0956797611430953CrossRefGoogle ScholarPubMed
Lakens, D. (2014). Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701–710. 10.1002/ejsp.2023CrossRefGoogle Scholar
Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362. 10.1177/1948550617697177CrossRefGoogle Scholar
Lakens, D. (2022). Sample size justification. Collabra: Psychology, 8(1),33267. 10.1525/collabra.33267CrossRefGoogle Scholar
Lakens, D., Scheel, A. M., Isager, P. M. (2018). Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2), 259–269. 10.1177/2515245918770963CrossRefGoogle Scholar
LeBel, E. P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., Smith, C. T. (2013). Psychdisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8(4), 424–432. 10.1177/1745691613491437CrossRefGoogle ScholarPubMed
Leys, C., Delacre, M., Mora, Y. L., Lakens, D., & Ley, C. (2019). How to classify, detect, and manage univariate and multivariate outliers, with emphasis on pre-registration. International Review of Social Psychology, 32(1).CrossRefGoogle Scholar
List, J. A., Shaikh, A. M., Xu, Y. (2019). Multiple hypothesis testing in experimental economics. Experimental Economics, 22(4), 773–793. 10.1007/s10683-018-09597-5CrossRefGoogle Scholar
Loken, E., Gelman, A. (2017). Measurement error and the replication crisis. Science, 355(6325), 584–585. 10.1126/science.aal3618CrossRefGoogle ScholarPubMed
MacCoun, R. J., & Perlmutter, S. (2017). Blind analysis as a correction for confirmatory bias in physics and in psychology. In Psychological science under scrutiny: Recent challenges and proposed solutions.Google Scholar
McCloskey, D. N., Ziliak, S. T. (1996). The standard error of regressions. Journal of Economic Literature, 34(1), 97–114.Google Scholar
Miguel, E. (2021). Evidence on research transparency in economics. Journal of Economic Perspectives, 35(3), 193–214. 10.1257/jep.35.3.193CrossRefGoogle Scholar
Necker, S. (2014). Scientific misbehavior in economics. Research Policy, 43(10), 1747–1759. 10.1016/j.respol.2014.05.002CrossRefGoogle Scholar
Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45(3), 137–141.CrossRefGoogle Scholar
O’Boyle, E. H. Jr, Banks, G. C., Gonzalez-Mulé, E. (2017). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43(2), 376–399. 10.1177/0149206314527133CrossRefGoogle Scholar
Ofosu, G. K., & Posner, D. N. (2021). Pre-analysis plans: An early stocktaking. In Perspectives on Politics, pp. 1–17. https://www.cambridge.org/core/journals/perspectives-on-politics/article/preanalysis-plans-an-earlystocktaking/94E7FAE76001C45A04E8F5E272C773CE.Google Scholar
Olken, B. A. (2015). Promises and perils of pre-analysis plans. Journal of Economic Perspectives, 29(3), 61–80. 10.1257/jep.29.3.61CrossRefGoogle Scholar
Page, L., Noussair, C. N., Slonim, R. (2021). The replication crisis, the rise of new research practices and what it means for experimental economics. Journal of the Economic Science Association, 7(2), 210–225. 10.1007/s40881-021-00107-7CrossRefGoogle Scholar
Romano, J. P., Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica, 73(4), 1237–1282. 10.1111/j.1468-0262.2005.00615.xCrossRefGoogle Scholar
Romano, J. P., Wolf, M. (2016). Efficient computation of adjusted p-values for resampling-based stepdown multiple testing. Statistics & Probability Letters, 113, 38–40. 10.1016/j.spl.2016.02.012CrossRefGoogle Scholar
Schafmeister, F. (2021). The effect of replications on citation patterns: Evidence from a large-scale reproducibility project. Psychological Science, 32(10), 1537–1548. 10.1177/09567976211005767CrossRefGoogle ScholarPubMed
Scheel, A. M., Schijen, M. R., Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science, 4(2),25152459211007467. 10.1177/25152459211007467CrossRefGoogle Scholar
Schooler, J. W. (2014). Metascience could rescue the replication crisis. Nature, 515(7525), 9–9. 10.1038/515009aCrossRefGoogle ScholarPubMed
Serra-Garcia, M., Gneezy, U. (2021). Nonreplicable publications are cited more than replicable ones. Science Advances, 7(21),eabd1705. 10.1126/sciadv.abd1705CrossRefGoogle ScholarPubMed
Swanson, N., Christensen, G., Littman, R., Birke, D., Miguel, E., Paluck, E. L., & Wang, Z. (2020). Research transparency is on the rise in economics. In In AEA Papers and Proceedings, vol. 110, pp. 61–65.CrossRefGoogle Scholar
Thissen, D., Steinberg, L., Kuang, D. (2002). Quick and easy implementation of the Benjamini–Hochberg procedure for controlling the false positive rate in multiple comparisons. Journal of Educational and Behavioral Statistics, 27(1), 77–83. 10.3102/10769986027001077CrossRefGoogle Scholar
Van’t Veer, A. E., Giner-Sorolla, R. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12. 10.1016/j.jesp.2016.03.004CrossRefGoogle Scholar
Wicherts, J. M., Veldkamp, C. L., Augusteijn, H. E., Bakker, M., Van Aert, R., & Van Assen, M. A. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. In Frontiers in psychology, p. 1832.CrossRefGoogle Scholar
Yamagishi, T., Yamagishi, M. (1994). Trust and commitment in the United States and Japan. Motivation and Emotion, 18(2), 129–166. 10.1007/BF02249397CrossRefGoogle Scholar
Young, C., Holsteen, K. (2017). Model uncertainty and robustness: A computational framework for multimodel analysis. Sociological Methods & Research, 46(1), 3–40. 10.1177/0049124115610347CrossRefGoogle Scholar
Ziliak, S. T., McCloskey, D. N. (2004). Size matters: The standard error of regressions in the American Economic Review. Journal of Socio-Economics, 33(5), 527–546. 10.1016/j.socec.2004.09.024Google Scholar
Supplementary material: File

Arpinon and Espinosa supplementary material

Arpinon and Espinosa supplementary material
Download Arpinon and Espinosa supplementary material(File)
File 997 KB