Hostname: page-component-745bb68f8f-b95js Total loading time: 0 Render date: 2025-02-04T18:02:07.005Z Has data issue: false hasContentIssue false
  • English
  • Français

A Kuhnian Critique of Psychometric Research on Peer Review

Published online by Cambridge University Press:  01 January 2022

Carole J. Lee
Get access Rights & Permissions [Opens in a new window]

Abstract

Psychometrically oriented researchers construe low interrater reliability measures for expert peer reviewers as damning for the practice of peer review. I argue that this perspective overlooks different forms of normatively appropriate disagreement among reviewers. Of special interest are Kuhnian questions about the extent to which variance in reviewer ratings can be accounted for by normatively appropriate disagreements about how to interpret and apply evaluative criteria within disciplines during times of normal science. Until these empirical-cum-philosophical analyses are done, it will remain unclear the extent to which low interrater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers.

Type
Norms of Science and Science Policy
Information
Philosophy of Science , Volume 79 , Issue 5 , December 2012 , pp. 859 - 870
Copyright
Copyright © The Philosophy of Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bailar, John C. 1991. “Reliability, Fairness, Objectivity and Other Inappropriate Goals in Peer Review.” Behavioral and Brain Sciences 14:137–38.CrossRefGoogle Scholar
Bakanic, Von, McPhail, Clark, and Simon, Rita J.. 1987. “The Manuscript Review and Decision-Making Process.” American Sociological Review 52:631–42.CrossRefGoogle Scholar
Beyer, Janice. 1978. “Editorial Policies and Practices among Leading Journals in Four Scientific Fields.” Sociological Quarterly 19:68–88.CrossRefGoogle Scholar
Bornstein, Robert F. 1991. “Manuscript Review in Psychology: Psychometrics, Demand Characteristics, and an Alternative Model.” Journal of Mind and Behavior 12:429–68.Google Scholar
Campanario, Juan Miguel. 1995. “Commentary: On Influential Books and Journal Articles Initially Rejected Because of Negative Referees’ Evaluations.” Science Communication 16:304–25.CrossRefGoogle Scholar
Chubin, Daryl E., and Hackett, Edward J.. 1990. Peerless Science: Peer Review and U.S. Science Policy. Albany, NY: SUNY Press.Google Scholar
Cicchetti, Domenic V. 1991. “The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation.” Behavioral and Brain Sciences 14:119–86.CrossRefGoogle Scholar
Cole, Stephen. 1992. Making Science: Between Nature and Society. Cambridge, MA: Harvard University Press.Google Scholar
Cole, Stephen, Cole, Jonathan R., and Simon, Gary. 1981. “Chance and Consensus in Peer Review.” Science 214:881–86.CrossRefGoogle ScholarPubMed
Fiske, Donald, and Fogg, Louis. 1990. “But the Reviewers Are Making Different Criticisms of My Paper! Diversity and Uniqueness in Reviewer Comments.” American Psychologist 45:591–98.CrossRefGoogle Scholar
Fletcher, Robert H., and Fletcher, Suzanne W.. 1997. “Evidence for the Effectiveness of Peer Review.” Science and Engineering Ethics 3:35–50.CrossRefGoogle Scholar
Frank, Erica. 1996. “Editors’ Requests of Reviewers: A Study and a Proposal.” Preventative Medicine 25:102–4.CrossRefGoogle ScholarPubMed
Gardner, Martin J., and Bond, Jane. 1990. “An Exploratory Study of Statistical Assessment of Papers Published in the British Medical Journal.” Journal of the American Medical Association 263:1355–57.Google ScholarPubMed
Godlee, Fiona, Gale, Catharine R., and Martyn, Christopher N.. 1998. “Effect on the Quality of Peer Review of Blinding Reviewers and Asking Them to Sign Their Reports: A Randomized Controlled Trial.” Journal of the American Medical Association 280:237–40.Google ScholarPubMed
Gore, Sheila M., and Jones, Gerald. 1992. “The Lancet's Statistical Review Process: Areas for Improvement by Authors.” Lancet 340:100–102.CrossRefGoogle ScholarPubMed
Gottfredson, Stephen D. 1978. “Evaluating Psychological Research Reports: Dimensions, Reliability, and Correlates of Quality Judgments.” American Psychologist 33:920–34.CrossRefGoogle Scholar
Grice, Paul. 1989. “Logic and Conversation.” In Studies in the Way of Words. Cambridge, MA: Harvard University Press.Google Scholar
Hargens, Lowell L. 1988. “Scholarly Consensus and Journal Rejection Rates.” American Sociological Review 53:139–51.CrossRefGoogle Scholar
Hargens, Lowell L.. 1990. “Variation in Journal Peer Review Systems.” Journal of the American Medical Association 263:1348–52.Google ScholarPubMed
Hargens, Lowell L., and Herting, Jerald R.. 1990a. “Neglected Considerations in the Analysis of Agreement among Journal Referees.” Scientometrics 19:91–106.CrossRefGoogle Scholar
Hargens, Lowell L., and Herting, Jerald R.. 1990b. “A New Approach to Referees’ Assessments of Manuscripts.” Social Science Research 19:1–16.CrossRefGoogle Scholar
Hargens, Lowell L., and Herting, Jerald R.. 2006. “Analyzing the Association between Referees’ Recommendations and Editors’ Decisions.” Scientometrics 67:15–26.CrossRefGoogle Scholar
Hopewell, Sally, Loudon, Kirsty, Clarke, Mike J., Oxman, Andrew D., and Dickersin, Kay. 2009. “Publication Bias in Clinical Trials due to Statistical Significance or Direction of Trial Results.” Cochrane Database of Systematic Reviews, no. 1.Google Scholar
Hull, David L. 1988. Science as a Process. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Jayasinghe, Upali W. 2003. “Peer Review in the Assessment and Funding of Research by the Australian Research Council.” PhD diss., University of Western Sydney.Google Scholar
Jayasinghe, Upali W., Marsh, Herbert W., and Bond, Nigel. 2003. “A Multilevel Cross-Classified Modelling Approach to Peer Review of Grant Proposals: The Effects of Assessor and Researcher Attributes on Assessor Ratings.” Journal of the Royal Statistical Society A 166:279–300.CrossRefGoogle Scholar
Justice, Amy C., Cho, Mildred K., Winker, Margaret A., Berlin, Jesse A., Rennie, Drummond, and the PEER Investigators. 1998. “Does Masking Author Identity Improve Peer Review Quality? A Randomized Controlled Trial.” Journal of the American Medical Association 280:240–42.Google ScholarPubMed
Kuhn, Thomas S. 1977. “Objectivity, Value Judgment, and Theory Choice.” In The Essential Tension: Selected Studies in Scientific Tradition and Change, ed. Kuhn, Thomas S., 320–39. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Lamont, Michèle. 2009. How Professors Think: Inside the Curious World of Academic Judgment. Cambridge, MA: Harvard University Press.CrossRefGoogle Scholar
Lee, Carole J. 2008. “Applied Cognitive Psychology and the ‘Strong Replacement’ of Epistemology by Normative Psychology.” Philosophy of the Social Sciences 38:55–75.CrossRefGoogle Scholar
Lee, Carole J.. 2012. “Incentivizing Procedural Objectivity: Community Response to Publication Bias.” Unpublished manuscript, University of Washington, Seattle.Google Scholar
Lee, Carole J., and Schunn, Christian D.. 2011. “Social Biases and Solutions for Procedural Objectivity.” Hypatia 26:352–73.CrossRefGoogle Scholar
Lee, Carole J., Sugimoto, Cassidy R., Zhang, Guo, and Cronin, Blaise. Forthcoming. “Bias in Peer Review.” Journal of the American Society for Information Science and Technology.Google Scholar
Lindsey, Duncan. 1978. The Scientific Publication System in Social Science. San Francisco: Jossey-Bass.Google Scholar
Marsh, Herbert W., and Ball, Samuel. 1989. “The Peer Review Process Used to Evaluate Manuscripts Submitted to Academic Journals: Interjudgmental Reliability.” Journal of Experimental Education 57:151–69.CrossRefGoogle Scholar
Marsh, Herbert W., Jayasinghe, Upali W., and Bond, Nigel W.. 2008. “Improving the Peer-Review Process for Grant Applications: Reliability, Validity, Bias, and Generalizability.” American Psychologist 63:160–68.CrossRefGoogle Scholar
Merton, Robert K., and Zuckerman, Harriet. 1971. “Institutional Patterns of Evaluation in Science, 1971.” In The Sociology of Science: Theoretical and Empirical Investigations, ed. Storer, Norman W., 460–96. Chicago: University of Chicago Press.Google Scholar
Neuliep, James W., and Crandall, Rick. 1990. “Editorial Bias against Replication Research.” Journal of Social Behavior and Personality 5:85–90.Google Scholar
Rust, John, and Golombok, Susan. 2009. Modern Psychometrics: The Science of Psychological Assessment. 3rd ed. New York: Routledge.Google Scholar
Scott, William A. 1974. “Interreferee Agreement on Some Characteristics Submitted to the Journal of Personality and Social Psychology.” American Psychologist 29:698–702.CrossRefGoogle Scholar
Suls, Jerry, and Martin, Renee. 2009. “The Air We Breathe: A Critical Look at Practices and Alternatives in the Peer-Review Process.” Perspectives on Psychological Science 4:40–50.CrossRefGoogle Scholar
van Rooyen, Susan, Godlee, Fiona, Evans, Stephen, Smith, Richard, and Black, Nick. 2010. “Effect of Blinding and Unmasking on the Quality of Peer Review.” Journal of the American Medical Association 280:234–37.Google Scholar
Weller, Ann C. 2001. Editorial Peer Review: Its Strengths and Weaknesses. Medford, NJ: American Society for Information Science and Technology.Google Scholar
Ziman, J. 1969. “Information, Communication, Knowledge.” Nature 224:318–24.CrossRefGoogle ScholarPubMed
Zuckerman, Harriet, and Merton, Robert K.. 1971. “Patterns of Evaluation in Science: Institutionalisation, Structure and Functions of the Referee System.” Minerva 9:66–100.CrossRefGoogle Scholar