Hostname: page-component-586b7cd67f-2brh9 Total loading time: 0 Render date: 2024-11-23T21:51:33.477Z Has data issue: false hasContentIssue false
  • English
  • Français

Replication Is for Meta-Analysis

Published online by Cambridge University Press:  25 May 2022

Samuel C. Fletcher
Samuel C. Fletcher*
Affiliation:
Department of Philosophy and Minnesota Center for Philosophy of Science, University of Minnesota, Twin Cities, USA
*
Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

The role or function of experimental and observational replication within empirical science has implications for how replication should be measured. Broadly, there seems to be consensus that replication’s central goal is to confirm or vouchsafe the reliability of scientific findings. I argue that if this consensus is correct, then most of the measures of replication used in the scientific literature are actually poor indicators of this reliability or confirmation. Only meta-analytic measures of replication align functionally with the goals of replication. I conclude by addressing some objections to meta-analysis.

Type
Symposia Paper
Information
Philosophy of Science , Volume 89 , Issue 5 , December 2022 , pp. 960 - 969
Copyright
© The Author(s), 2022. Published by Cambridge University Press on behalf of the Philosophy of Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Baker, Monya. 2016. “1,500 Scientists Lift the Lid on Reproducibility.” Nature 533 (7604):452–54.CrossRefGoogle ScholarPubMed
Glenn, Begley, C., and Ellis, Lee M.. 2012. “Raise Standards for Preclinical Cancer Research: Drug Development.” Nature 483 (7391):531–33.Google Scholar
Borenstein, Michael, Hedges, Larry V., Higgins, Julian P. T., and Rothstein, Hannah R.. 2021. Introduction to Meta-Analysis. 2nd ed. Oxford: John Wiley & Sons.CrossRefGoogle Scholar
Braver, Sanford L., Thoemmes, Felix J., and Rosenthal, Robert. 2014. “Continuously Cumulating Meta-Analysis and Replicability.” Perspectives on Psychological Science 9 (3):333–42.CrossRefGoogle ScholarPubMed
Bruner, Justin P., and Holman, Bennett. 2019. “Self-Correction in Science: Meta-Analysis, Bias and Social Structure.” Studies in History and Philosophy of Science 78:93–97.CrossRefGoogle ScholarPubMed
Camerer, Colin F, Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, et al. 2016. “Evaluating Replicability of Laboratory Experiments in Economics.” Science 351 (6280):1433–36.CrossRefGoogle ScholarPubMed
Camerer, Colin F., Anna Dreber, Felix Holzmeister, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, et al. 2018. “Evaluating the Replicability of Social Science Experiments in Nature and Science Between 2010 and 2015.” Nature Human Behaviour 2 (9):637–44.CrossRefGoogle ScholarPubMed
Carter, Evan C., Schönbrodt, Felix D., Gervais, Will M., and Hilgard, Joseph. 2019. “Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods.” Advances in Methods and Practices in Psychological Science 2 (2):115–44.CrossRefGoogle Scholar
Cartwright, Nancy. 1991. “Replicability, Reproducibility, and Robustness: Comments on Harry Collins.” History of Political Economy 23 (1):143–55.CrossRefGoogle Scholar
Crupi, Vincenzo. 2021. “Confirmation.” In The Stanford Encyclopedia of Philosophy, edited by Edward, N. Zalta. Stanford: Stanford University Press.Google Scholar
Feest, Uljana. 2019. “Why Replication is Overrated.” Philosophy of Science 86 (5):895–905.CrossRefGoogle Scholar
Fidler, Fiona, and Wilcox, John. 2018. “Reproducibility of Scientific Results.” In The Stanford Encyclopedia of Philosophy, edited by Edward, N. Zalta. Stanford: Stanford University Press.Google Scholar
Fletcher, Samuel C. 2021a. “The Role of Replication in Psychological Science.” European Journal for Philosophy of Science 11 (23):1–19.CrossRefGoogle Scholar
Fletcher, Samuel C. 2021b. “How (Not) to Measure Replication.” European Journal for Philosophy of Science 11 (57):1–27.CrossRefGoogle Scholar
Fletcher, Samuel C., and Mayo-Wilson, Conor. Forthcoming. “Evidence in Classical Statistics.” In Routledge Handbook of the Philosophy of Evidence, edited by Lasonen-Aarnio, Maria and Littlejohn, Clayton. London: Routledge.Google Scholar
Gómez, Omar S., Juristo, Natalia, and Vegas, Sira. 2010. “Replications Types in Experimental Disciplines.” In Proceedings of the 2010 ACM-IEEE international symposium on empirical software engineering and measurement, ESEM ’10, 1–10. New York: Association for Computing Machinery.CrossRefGoogle Scholar
Holman, Bennett. 2019. “In Defense of Meta-Analysis.” Synthese 196 (8):3189–211.CrossRefGoogle Scholar
Jukola, Saana. 2015. “Meta-Analysis, Ideals of Objectivity, and the Reliability of Medical Knowledge.” Science & Technology Studies 28 (3):101–20.CrossRefGoogle Scholar
Klein, Richard A. Michaelangelo Vianello, Fred Hasselman, Adams, Byron G., Reginald, B. Adams Jr., Sinan Alper, Aveyard, Mark, et al. 2018. “Many Labs 2: Investigating Variation in Replicability Across Samples and Settings.” Advances in Methods and Practices in Psychological Science 1 (4):443–90.10.1177/2515245918810225CrossRefGoogle Scholar
Kvarven, Amanda, Strømland, Eirik, and Johannesson, Magnus. 2020. “Comparing Meta-Analyses and Preregistered Multiple-Laboratory Replication Projects.” Nature Human Behaviour 4 (4):423–34.CrossRefGoogle ScholarPubMed
Leonelli, Sabina. 2018. “Rethinking Reproducibility as a Criterion for Research Quality.” In Including a Symposium on Mary Morgan: Curiosity, Imagination, and Surprise (Volume 36B of Research in the History of Economic Thought and Methodology), edited by Boumans, Marcel and Chao, Hsiang-Ke, 129–46. Bingley: Emerald Publishing Ltd.Google Scholar
Machery, Edouard. 2020. “What Is a Replication?” Philosophy of Science 87 (4):545–67.CrossRefGoogle Scholar
Mayo, Deborah G. 1996. Error and the Growth of Experimental Knowledge. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Mayo, Deborah G. 2018. Statistical Inference as Severe Testing. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Maxwell, Scott E., Lau, Micahel Y., and Howard, George S.. 2015. “Is Psychology Suffering from a Replication Crisis? What Does ‘Failure to Replicate’ Really Mean?” American Psychologist 70 (6):487–98.CrossRefGoogle ScholarPubMed
Mobley, Aaron, Linder, Suzanne K., Russell Braeuer, Lee M. Ellis, and Zwelling, Leonard. 2013. “A Survey on Data Reproducibility in Cancer Research Provides Insights into Our Limited Ability to Translate Findings from the Laboratory to the Clinic.” PLoS ONE 8 (5):e63221.CrossRefGoogle ScholarPubMed
Moonesinghe, Ramal, Khoury, Muin J., and Cecile, A.. Janssens, J. W.. 2007. “Most Published Research Findings Are False—but a Little Replication Goes a Long Way.” PLoS Medicine 4 (2):e28.CrossRefGoogle ScholarPubMed
Nosek, Brian A., and Errington, Timothy M.. 2020. “What is Replication?” PLoS Biology 18 (3):e3000691.CrossRefGoogle ScholarPubMed
Open Science Collaboration (OSC). 2015. “Estimating the Reproducibility of Psychological Science.” Science 349 (6251):943–51.Google Scholar
Popper, Karl R. 1959. The Logic of Scientific Discovery. London: Routledge.Google Scholar
Romero, Felipe. 2016. “Can the Behavioral Sciences Self-Correct? A Social Epistemic Study.” Studies in History and Philosophy of Science 60:55–69.CrossRefGoogle ScholarPubMed
Romero, Felipe. 2019. “Philosophy of Science and the Replicability Crisis.” Philosophy Compass 14 (11):e12633.CrossRefGoogle Scholar
Schmidt, Stefan. 2009. “Shall We Really Do It Again? The Powerful Concept of Replication Is Neglected in the Social Sciences.” Review of General Psychology 13 (2):90–100.CrossRefGoogle Scholar
Schmidt, Stefan. 2017. “Replication.” In Toward a More Perfect Psychology: Improving Trust, Accuracy, and Transparency in Research, edited by Matthew, C. Makel and Jonathan, A. Plucker, 233–53. Washington, DC: American Psychological Association.CrossRefGoogle Scholar
Schmidt, Frank L., and Hunter, John E.. 2015. Methods of Meta-Analysis: Correcting Error and Bias in Research Findings, 3rd ed. Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
Simons, Daniel J. 2014. “The Value of Direct Replication.” Perspectives on Psychological Science 9 (1):76–80.CrossRefGoogle ScholarPubMed
Spellman, Barbara A. (2015). “A Short (Personal) Future History of Revolution 2.0.” Perspectives on Psychological Science 10 (6):886–99.CrossRefGoogle Scholar
Stegenga, Jacob. (2011). “Is Meta-Analysis the Platinum Standard of Evidence?” Studies in History and Philosophy of Biological and Biomedical Sciences 42 (4):497–507.CrossRefGoogle ScholarPubMed