Hostname: page-component-586b7cd67f-rdxmf Total loading time: 0 Render date: 2024-11-27T23:46:14.996Z Has data issue: false hasContentIssue false
  • English
  • Français

Peer-review practices of psychological journals: The fate of published articles, submitted again

Published online by Cambridge University Press:  04 February 2010

Douglas P. Peters  and
Stephen J. Ceci
Douglas P. Peters
Affiliation:
Department of Psychology, University of North Dakota, Grand Forks, N.D. 58202
Stephen J. Ceci
Affiliation:
Department of Human Development and Family Studies, Cornell University, Ithaca, N.Y. 74853
Get access Rights & Permissions [Opens in a new window]

Abstract

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.

The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.

With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated.

Keywords

biasevaluationjournal review systemmanuscript reviewpeer reviewpublication practicesratingsrefereeingreliabilityscience management
Type
Target Article
Information
Behavioral and Brain Sciences , Volume 5 , Issue 2: Special Symposium Issue , June 1982 , pp. 187 - 195
Copyright
Copyright © Cambridge University Press 1982

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abelson, P. H. (1979) Problems of science faculties (editorial). Science 204:133. [CM]CrossRefGoogle Scholar
Adair, R. K. (1981) Anonymous refereeing. Physics Today 34:13–15. [RAG]CrossRefGoogle Scholar
American Psychological Association (1963–1969) Reports of Project on Scientific Information Exchange in Psychology. Vols. 1–3. Washington, D.C.: APA, [BCG]Google Scholar
American Psychological Association (1972) Eight APA journals initiate controversial blind reviewing. APA Monitor 3:5. [taDPP]Google Scholar
American Psychological Association (1973) Ethical principles in the conduct of research with human participants. American Psychologist 28:79–80. [JLF, BM]CrossRefGoogle Scholar
American Psychological Association (1980) Summary report of journal operations for 1979. American Psychologist 35:575. [taDPP]CrossRefGoogle Scholar
Anastasi, A. (1958) Differential psychology. 3rd ed.New York: Macmillan. [GJW]Google Scholar
Armstrong, J. S. (1979) Advocacy and objectivity in science. Management Science 25:423–28. [JSA]CrossRefGoogle Scholar
Armstrong, J. S. (1980a) Advocacy as a scientific strategy: The mitroff myth. Academy of Management Review 5:509–11. [JSA]CrossRefGoogle Scholar
Armstrong, J. S. (1980b) Unintelligible management research and academic prestige. Interfaces 10:80–86. [JSA]CrossRefGoogle Scholar
Armstrong, J. S. (1982, forthcoming) Research on scientific journals: Implications for editors and authors. Journal of Forecasting and Medical Hypotheses. [JSA]CrossRefGoogle Scholar
Bartko, J. J. (1966) The intraclass correlation coefficient as a measure of reliability. Psychological Reports 19:3–11. [GJW]CrossRefGoogle ScholarPubMed
Bartley, W. W. (1962) The retreat to commitment. New York: Alfred A. Knopf. [MJM]Google Scholar
Benwell, R. (1979) Authors anonymous? Physics Bulletin 30:288. [RAG]CrossRefGoogle Scholar
Bernard, H. R. (1980) CARS: Computer assisted referee selection. Journal of Research Communication Studies 2:149–57. [HRB]Google Scholar
Beyer, J. (1978) Editorial policies and practices among leading journals in four scientific fields. Sociological Quarterly 19:68–88. [JMB]CrossRefGoogle Scholar
Blissett, M. (1972) Politics in science. Boston: Little, Brown. [MB]Google Scholar
Boffey, P. M. (1975) The brain bank of America: An inquiry into the politics of science. New York: McGraw-Hill. [CM]Google Scholar
Bowen, D. D.; Perloff, R. & Jacoby, J. (1972) Improving manuscript evaluation procedures. American Psychologist 27:221–25. [taDPP]CrossRefGoogle Scholar
Brackbill, Y. & Korton, F. (1970) Journal reviewing practices: Authors' and APA members' suggestions for revision. American Psychologist 25:937–40. [taDPP]CrossRefGoogle Scholar
Broad, W. J. (1980a) Imbroglio at Yale. 1. Emergence of a fraud. Science 210:38–11. [DdB]CrossRefGoogle Scholar
Broad, W. J. (1980b) Would-be academician pirates papers. Science 208:1438–10. [DdB]CrossRefGoogle ScholarPubMed
Broad, W. J. (1981a) Congress told fraud issue “exaggerated.” Science 212:421. [CM]CrossRefGoogle ScholarPubMed
Broad, W. J. (1981b) Fraud and the structure of science. Science 212:137–41. [DdB, CM]CrossRefGoogle ScholarPubMed
Broad, W. J. (1981c) The publishing game: Getting more for less. Science 211:1137–39. [DdB]CrossRefGoogle ScholarPubMed
Bronfenbrenner, U. (1977) Toward an experimental ecology of human development. American Psychologist 32:513–31. [taDPP]CrossRefGoogle Scholar
Bruner, J. S. (1962) On knowing: Essays for the left hand. Cambridge, Mass.: Harvard University Press. [IIM]CrossRefGoogle Scholar
California may be sued on secret files. (1978) Times Higher Education Supplement, 11 3, p. 5. [CM]Google Scholar
Campbell, D. T. & Fiske, D. W. (1959) Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin 56:81–105. [rDPP]CrossRefGoogle ScholarPubMed
Caplow, T. & McGee, R. J. (1958) The academic marketplace. New York: Basic Books. [CM]Google Scholar
Carey, W. D. (1975) Peer review revisited. Science 189:331. [MB]CrossRefGoogle Scholar
Carta, D. G. (1978) Forum for rejected papers. IEEE Spectrum 15:13. [RAG]Google Scholar
Cherfas, J. (1980) Only the names have been changed to protect … whom? New Scientist, 20 03, p. 950. [Ed.]Google Scholar
Chubin, D. E. (1980) Competence is not enough. Contemporary Sociology 9:204–7. [DEC, CM]CrossRefGoogle Scholar
Chubin, D. E. & Connolly, T. (1982) Research trails and science policies: Local and extra-local negotiation of scientific work. In: Scientific establishments and hierarchies, ed. Elias, N., Martins, H., & Whitley, H.. Sociology of the Sciences, vol. 6, pp. 293–311. [ALP]CrossRefGoogle Scholar
Cicchetti, D. V. (1980) Reliability of reviews for the American Psychologist: A biostatistical assessment of the data. American Psychologist 35:300–303. [DVC, tarDPP, GJW]CrossRefGoogle Scholar
Cicchetti, D. V. & Conn, H. O. (1976) A statistical analysis of reviewer agreement and bias in evaluating medical abstracts. Yale Journal of Biology and Medicine 49:373–83. [taDPP]Google Scholar
Cicchetti, D. V. & Eron, L. D. (1979) The reliability of manuscript reviewing for the Journal of Abnormal Psychology. 1979 Proceedings of the Social Statistics Section, pp. 596–600. Washington, D.C.: American Statistics Association. [DVC, tarDPP]Google Scholar
Coe, R. K. & Weinstock, I. (1967) Editorial policies of major economic journals. Quarterly Review of Economics and Business 7:37–43. [JSA]Google Scholar
Cohen, J. (1968) Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70:213–20. [GJW]CrossRefGoogle ScholarPubMed
Cole, J. R. & Cole, S. (1972) The Ortega hypothesis: Citation analysis suggests that only a few scientists contribute to scientific progress. Science 178:368–74. [CM, taDPP]CrossRefGoogle Scholar
Cole, J. R. & Cole, S. (1973) Social stratification in science. Chicago: University of Chicago Press. [CM]Google Scholar
Cole, J. R. & Cole, S. (1981) Peer review in the National Science Foundation: Phase II of a study. Washington, D.C.: National Academy of Sciences. [Ed.]Google Scholar
Cole, S.; Cole, J. R. & Simon, G. A. (1981) Chance and consensus in peer review. Science 214:881–86. [Ed., rDPP]CrossRefGoogle ScholarPubMed
Cole, S.; Rubin, L. & Cole, J. R. (1977) Peer review and the support of science. Scientific American 237:34–11. [RTL, CM]CrossRefGoogle ScholarPubMed
Cole, S.; Rubin, L. & Cole, J. R. (1978) Peer review in the National Science Foundation: Phase 1 of a study. Washington, D.C.: National Academy of Sciences. [DEC, CM]Google Scholar
Collins, H. M. (1981) Stages in the empirical programme of relativism. Social Studies of Science 11:3–10. [MDG]CrossRefGoogle Scholar
Colman, A. M. (1979) Editorial role in author-referee disagreements. Bulletin of the British Psychological Society 32:390–91. [AMC, taDPP]Google Scholar
Colman, A. M. (1981) What is psychology? London: Kogan Page. [AMC]Google Scholar
Cone, J. D. & Foster, S. L. (in press) Direct observation in clinical psychology. In: Handbook of research methods in clinical psychology, ed. P. C. Kendall & J. N. Butcher. New York: John Wiley & Sons. [JDC]Google Scholar
Cook, T. D. & Campbell, D. T. (1979) Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally. [rDPP]Google Scholar
Cox, W. M. & Catt, V. (1977) Productivity ratings of graduate programs in psychology based on publication in the journals of the American Psychological Association. American Psychologist 32:793–813. [taDPP]CrossRefGoogle Scholar
Crandall, R. (1977) How qualified are editors? American Psychologist 32:578–79. [RC]CrossRefGoogle Scholar
Crandall, R. (1978a) Interrater agreement on manuscripts is not so bad! American Psychologist 33:623–24. [RC, taDPP]CrossRefGoogle Scholar
Crandall, R. (1978b) The relationship between quantity and quality of publications. Personality and Social Psychology Bulletin 4:379–80. [RC]CrossRefGoogle Scholar
Crandall, R. & Diener, E. (1978) Determining authorships of scientific papers. Drug Intelligence and Clinical Pharmacy 12:375. [RC]CrossRefGoogle Scholar
Crane, D. (1967) The gatekeepers of science: Some factors affecting the selection of articles for scientific journals. American Sociologist 32:195–201. [MDG, MJM, taDPP]Google Scholar
Crane, D. (1972) Invisible colleges. Chicago: University of Chicago Press. [DLE]Google Scholar
Davidson, J. M. & Davidson, R. J., eds. (1980) The psychobiology of consciousness. New York: Plenum. [MJM]CrossRefGoogle Scholar
DeBakey, L. (1976) Reviewing. In: The scientific journal: Editorial policies and practices, pp. 1–23. St. Louis: C. V. Mosby Company. [LD]Google Scholar
DeBakey, L. (1978) Communication, biomedical: II. Scientific publishing. In: Encyclopedia of bioethics, ed. Reich, W. T., vol. 1, pp. 188–94. New York: Free Press. [LD]Google Scholar
DeBakey, L. & DeBakey, S. (1976) Impartial, signed reviews. New England Journal of Medicine 294:564. [LD]Google ScholarPubMed
Diener, E. & Crandall, R. (1978) Ethics in social and behavioral research. Chicago: University of Chicago Press. [RC]Google Scholar
Eckberg, D. & Hill, L. (1979) The paradigm concept and sociology. American Sociological Review 44:925–37. [DLE]CrossRefGoogle Scholar
Editorial note. (1965) Helgolander Wissenschaftliche Meeresuntersuchen 12 (07): 218. [CM]Google Scholar
Editoral. (1979) Physical Review Letters 43. [RKA]Google Scholar
Endler, N. S.; Rushton, J. P. & Roediger, H. L. (1978) Productivity and scholarly impact (citations) of British, Canadian, and U.S. departments of psychology. American Psychologist 33:1064–82. [taDPP]CrossRefGoogle Scholar
Freese, L. (1979) On changing some role relationships in the editorial review process. American Sociologist 14:231–38. [DEC, rDPP]Google Scholar
Garfield, E. (1979a) Citation indexing: Its theory and application in science, technology, and humanities. New York: John Wiley & Sons. [taDPP, MJW]Google Scholar
Garfield, E. (1979b) Journal citation report; A bibliometric analysis of social science journals in the ISI data base. Philadelphia: Institute for Scientific Information. [taDPP, MJW]Google Scholar
Garvey, W. D. & Griffith, B. C. (1964) Scientific information exchange in psychology. Science 146:1655–59. [BCG]CrossRefGoogle ScholarPubMed
Garvey, W. D. & Griffith, B. C. (1971) Scientific communication: Its role in the conduct of research and creation of knowledge. American Psychologist 26:349–62. [DdB, BCG, taDPP]CrossRefGoogle Scholar
Garvey, W.; Lin, N. & Nelson, C. (1970) Communication in the physical and social sciences. Science 170:1166–73. [BCG]CrossRefGoogle Scholar
Gibbs, J. C. (1979) The meaning of ecologically oriented inquiry in contemporary psychology. American Psychologist 34:127–40. [taDPP]CrossRefGoogle Scholar
Glenn, N. (1976) The journal article review process: Some proposals for change. American Sociologist 11:179–85. [DEC]Google Scholar
Goodstein, L. D. & Brazis, K. L. (1970) Credibility of psychologists: An empirical study. Psychological Reports 27:835–38. [JSA, taDPP]CrossRefGoogle Scholar
Gordon, M. D. (1980) The role of referees in scientific communication. In: The psychology of written communication, ed. Hartley, J., pp. 263–75. London: Kogan Page. [MDG, JH, RMP, tarDPP, SP]Google Scholar
Gordon, R. A. (1980) The advantages of a simple system of optional published refereeing. Speculations in Science and Technology 3:607–9. [RAG]Google Scholar
Gottfredson, S. D. (1978) Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments. American Psychologist 33:920–34. [RO, taDPP]CrossRefGoogle Scholar
Gove, W. R. (1979) The review process and its consequences in the major sociology journals. Contemporary Sociology 8:799–804. [taDPP]CrossRefGoogle Scholar
Greenberg, D. S. (1980) Scams and sleaze in science. Clinical Chemistry News 6:5. [DEC]Google Scholar
Griffith, B. & Small, H. (1976) A Philadelphia study of the structure of science: The structure of the social and behavioral sciences' literature. In: Proceedings, First International Conference on Social Studies of Science. Ithaca, N.Y.: Society for the Social Studies of Science. [BCG]Google Scholar
Hagstrom, W. O. (1974) Competition in science. American Sociological Review 39:1–18. [CM]CrossRefGoogle Scholar
Hall, J. (1979) Author review of reviewers. American Psychologist 34:798. [taDPP, GJW]CrossRefGoogle Scholar
Hargens, L. (1975) Patterns of scientific research: A comparative analysis of research in three scientific fields. Washington, D.C.: American Sociological Association. [JMB]Google Scholar
Harnad, S. (1979) Creative disagreement. Sciences 19:18–20. [DVC, Ed., WMH, taDPP]CrossRefGoogle Scholar
Harnad, S. (1982) Rational disagreement in peer review. (Submitted for publication.) [Ed.]Google Scholar
Hartmann, D. P. & Wood, D. D. (1981) Observational methods. In: International handbook of behavior modification and therapy, ed. Kazdin, A. E.. New York: Plenum. [JDC]Google Scholar
Hawkins, R. G.; Ritter, L. S. & Walter, I. (1973) What economists think of their journals. Journal of Political Economy 81:1017–32. [JSA]CrossRefGoogle Scholar
Hendrick, C. (1976) Editorial comment. Personality and Social Psychology Bulletin 2:207–8. [GJW]CrossRefGoogle Scholar
Hendrick, C. (1977) Editorial comment. Personality and Social Psychology Bulletin 3:1–2. [taDPP]CrossRefGoogle Scholar
Hensler, D. R. (1976) Perceptions of the National Science Foundation peer review process: A report on a survey of NSF reviewers and applicants. NSF publication #77–33. [RTL, CM]Google Scholar
Herrnstein, R. J. (1977) Doing what comes naturally: A reply to Professor Skinner. American Psychologist 32:1013–16. [taDPP]CrossRefGoogle Scholar
Holden, C. (1979) Ethics and social science research. Science 206:357–340. [rDPP]Google ScholarPubMed
Holden, C. (1980) Not what you know, but where you're from. Science 209:1097. [SP]CrossRefGoogle Scholar
Honig, W. (1980a) Concluding anti-relativity. Speculations in Science and Technology 3:361–63. [WMH]Google Scholar
Honig, W. (1980b) Editorials on our evolving policy: Opening statements; Further statements on speculation; “They laughed at Columbus” and other author syndromes; Our first year, Modesty and age “paradigms”; Einstein centennial issue – Alternates to special relativity; Mathematics in physical science, or why the tail wags the dog; Comment on submissions; Some additional thoughts on speculation; Anti-relativity. Speculations in Science and Technology 3:233–43. [WMH]Google Scholar
Honig, W. (1980c) The review process: Before, after and during. Speculations in Science and Technology 3:513–16. [WMH]Google Scholar
Horn, R. E. (1980) Results with structured writing using the information mapping writing service standards. Paper available from the author, Information Resources, Inc., 133 Massachusetts Avenue, Lexington, MA 02173. [JH]Google Scholar
Horrobin, D. F. (1974) Referees and research administrators: Barriers to scientific research? British Medical Journal 2:216–18. [CM]CrossRefGoogle ScholarPubMed
Huxley, L. (1900) Life and letters of Thomas Henry Huxley. London: Macmillan and Co. [LD]Google Scholar
Hagstrom, W. O. (1974) Competition in science. American Sociological Review 39:1–18. [CM]CrossRefGoogle Scholar
Hall, J. (1979) Author review of reviewers. American Psychologist 34:798. [taDPP, GJW]CrossRefGoogle Scholar
Hargens, L. (1975) Patterns of scientific research: A comparative analysis of research in three scientific fields. Washington, D.C.: American Sociological Association. [JMB]Google Scholar
Harnad, S. (1979) Creative disagreement. Sciences 19:18–20. [DVC, Ed., WMH, taDPP]CrossRefGoogle Scholar
Harnad, S. (1982) Rational disagreement in peer review. (Submitted for publication.) [Ed.]Google Scholar
Hartmann, D. P. & Wood, D. D. (1981) Observational methods. In: International handbook of behavior modification and therapy, ed. Kazdin, A. E.. New York: Plenum. [JDC]Google Scholar
Hawkins, R. G.; Ritter, L. S. & Walter, I. (1973) What economists think of their journals. Journal of Political Economy 81:1017–32. [JSA]CrossRefGoogle Scholar
Hendrick, C. (1976) Editorial comment. Personality and Social Psychology Bulletin 2:207–8. [GJW]CrossRefGoogle Scholar
Hendrick, C. (1977) Editorial comment. Personality and Social Psychology Bulletin 3:1–2. [taDPP]CrossRefGoogle Scholar
Hensler, D. R. (1976) Perceptions of the National Science Foundation peer review process: A report on a survey of NSF reviewers and applicants. NSF publication #77–33. [RTL, CM]Google Scholar
Herrnstein, R. J. (1977) Doing what comes naturally: A reply to Professor Skinner. American Psychologist 32:1013–16. [taDPP]CrossRefGoogle Scholar
Holden, C. (1979) Ethics and social science research. Science 206:357–340. [rDPP]Google ScholarPubMed
Holden, C. (1980) Not what you know, but where you're from. Science 209:1097. [SP]CrossRefGoogle Scholar
Honig, W. (1980a) Concluding anti-relativity. Speculations in Science and Technology 3:361–63. [WMH]Google Scholar
Honig, W. (1980b) Editorials on our evolving policy: Opening statements; Further statements on speculation; “They laughed at Columbus” and other author syndromes; Our first year, Modesty and age “paradigms”; Einstein centennial issue – Alternates to special relativity; Mathematics in physical science, or why the tail wags the dog; Comment on submissions; Some additional thoughts on speculation; Anti-relativity. Speculations in Science and Technology 3:233–43. [WMH]Google Scholar
Honig, W. (1980c) The review process: Before, after and during. Speculations in Science and Technology 3:513–16. [WMH]Google Scholar
Horn, R. E. (1980) Results with structured writing using the information mapping writing service standards. Paper available from the author, Information Resources, Inc., 133 Massachusetts Avenue, Lexington, MA 02173. [JH]Google Scholar
Horrobin, D. F. (1974) Referees and research administrators: Barriers to scientific research? British Medical Journal 2:216–18. [CM]CrossRefGoogle ScholarPubMed
Huxley, L. (1900) Life and letters of Thomas Henry Huxley. London: Macmillan and Co. [LD]Google Scholar
Ingelfinger, F. J. (1974) Peer review in biomedical publication. American Journal of Medicine 56:686–92. [taDPP]CrossRefGoogle ScholarPubMed
Jones, R. (1974) Rights, wrongs and referees. New Scientist 61:758–59. [taDPP]Google Scholar
Kelly, W. (1972) Pogo: We have met the enemy and he is us. New York: Simon & Schuster. [DVC]Google Scholar
Kerr, S.; Tolliver, J. & Petree, D. (1977) Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal 20:132–41. [JSA]CrossRefGoogle Scholar
Koestler, A. (1971) The case of the midwife toad. London: Hutchinson. [CM]Google Scholar
Korten, F. & Griffith, B. (ca. 1970) Editorial review in psychological journals: A survey of authors, editors, and readers. Washington, D. C.: American Psychological Association. [BCG]Google Scholar
Kosinski, J. (1968) Steps. New York: Random House. [CR]Google Scholar
Koulack, D. & Keselman, H. J. (1975) Ratings of psychology journals by members of the American Psychological Association. American Psychologist 30:1049–53. [taDPP]CrossRefGoogle Scholar
Kuhn, T. S. (1970) The structure of scientific revolutions. 2d ed., enl. Chicago: University of Chicago Press. [DLE, BM]Google Scholar
Kuhn, T. S. (1974) Second thoughts on paradigms. In: The structure of scientific theories, ed. Suppe, F., pp. 459–99. Urbana: University of Illinois Press. [DLE]Google Scholar
Kumar, K. (1979) Optional published refereeing. Physics Today 32:13–14. [RAG]Google Scholar
Lakatos, I. (1970) Falsification and the methodology of scientific research programmes. In: Criticism and the growth of knowledge, ed. Lakatos, I. & Musgrave, A., pp. 91–196. Cambridge: Cambridge University Press. [MJM]CrossRefGoogle Scholar
Lakatos, I. & Musgrave, A., eds. (1970) Criticism and the growth of knowledge. Cambridge: Cambridge University Press. [MJM]CrossRefGoogle Scholar
Lazarus, D. (1980) Changes in “The Physical Review” and “Physical Review Letters.” Physical Review Letters 45:1605–6. [RAG]CrossRefGoogle Scholar
Leopold, A. C. (1978) The act of creation: Creative processes in science. BioScience 28:436–40. [CM]CrossRefGoogle Scholar
Levenson, H.; Burford, B.; Bonno, B. & Davis, L. (1975) Are women still prejudiced against women? A replica and extension of Goldberg's study. Journal of Psychology 89:67–71. [RO]CrossRefGoogle Scholar
Lewis, L. S. (1972) Academic freedom cases and their disposition. Change 4:8, 77. [CM]CrossRefGoogle Scholar
Lewis, L. S. (1975) Scaling the ivory tower: Merit and its limits in academic careers. Baltimore: Johns Hopkins University Press. [CM]Google Scholar
Lindsey, D. (1978) The scientific publication system in social science. San Francisco: Jossey-Bass. [DEC, rDPP, RO]Google Scholar
Lovas, S. (1980) Higher degree examination procedures in Australian universities. Vestes (Federation of Australian University Staff Associations) 23:10–20. [CM]Google Scholar
McCall, R. B. (1977) Challenges to a science of developmental psychology. Child Development 48:333–44. [taDPP]CrossRefGoogle Scholar
McCartney, J. L. (1973) Manuscript reviewing. Sociological Quarterly 14:290, 444–46. [RC, taDPP]Google Scholar
McCutchen, C. (1976) An evolved conspiracy. New Scientist 70:225. [taDPP]Google Scholar
McGuigan, F. J. (1968) Experimental psychology: A methodological approach. 2d ed.Englewood Cliffs, N.J.: Prentice-Hall. [BM]Google Scholar
McGuire, W. J. (1973) The yin and yang of social psychology: Seven koan. Journal of Personality and Social Psychology 26:446–56. [taDPP]CrossRefGoogle Scholar
McReynolds, P. (1971) Reliability of ratings of research papers. American Psychologist 26:400–101. [DP, taDPP]CrossRefGoogle Scholar
Mahoney, M. J. (1976) Scientist as subject: The psychological imperative. Cambridge, Mass.: Ballinger. [MJM, CM, RO]Google Scholar
Mahoney, M. J. (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive. Therapy and Research 1:161–75. [JSA, MJM, RO, taDPP]CrossRefGoogle Scholar
Mahoney, M. J. (1979) Psychology of the scientist: An evaluative review. Social Studies of Science 9:349–75. [JSA, DEC, CM]CrossRefGoogle Scholar
Mahoney, M. J. (1982) Psychotherapy and human change processes. In: Psychotherapy re- search and behavior change, ed. American Psychological Association. Washington, D.C.: American Psychological Association. [MJM]Google Scholar
Mahoney, M. J. (in press) Clinical psychology and scientific inquiry. International Journal of Psychology. [MJM]Google Scholar
Mahoney, M. J.; Kazdin, A. E. & Kenigsberg, M. (1978) Getting published: The effects of self-citation and institutional affiliation. Cognitive Therapy and Research 2:69–70. [JSA, MJM, RO, SP]CrossRefGoogle Scholar
Mahoney, M. J. & Kimper, T. P. (1976) From ethics to logic: A survey of scientists. In: Scientist as subject, Mahoney, M. J., ed. pp. 187–93. Cambridge, Mass.: Ballinger. [JSA]Google Scholar
Manwell, C. (1979) Peer review: A case history from the Australian Research Grants Committee. Search (ANZAAS) 10:81–86. [CM]Google Scholar
Manwell, C. (1981) An open letter to the president of FAUSA. Australian Higher Education Supplement, ed. J. Bremer, 05 27. [CM]Google Scholar
Manwell, C. & Baker, C. M. A. (1979) The double helix: Science and myth in the act of creation. BioScience 29:742–46. [CM]CrossRefGoogle Scholar
Manwell, C. & Baker, C. M. A. (1981) Honesty in science: A partial test of a sociobiological model of the social structure of science. Search (ANZAAS) 12:151–60. [CM]Google Scholar
Margulis, L. (1977) Peer review attacked (letter). The Sciences 17:5, 31. [CM]Google Scholar
Markle, A. & Rjnn, R. C. (1977) Author's guide to journals in psychology, psychiatry, if social work. New York: Haworth Press. [taDPP]Google Scholar
Martin, B. (1979) The bias of science. Society for Social Responsibility in Science, P. O. Box 48, O'Connor, A. C. T., Australia 2601. [CM]Google Scholar
Martin, B. (1981a) The dismissal of Dr. M. E. Spautz from the University of Newcastle (New South Wales). Unpublished manuscript, availabe from Dr. Brian Martin, Dept. of Applied Mathematics, School of General Studies, Australian National University, Canberra, A. C. T. 2600. [CM]Google Scholar
Martin, B. (1981b) The scientific straightjacket: The power structure of science and the suppression of environmental scholarship. Ecologist 11:33–43. [CM]Google Scholar
Merton, R. K. (1968a) The Matthew Effect in science. Science 159:56–63. [DdB, CM]CrossRefGoogle Scholar
Merton, R. K. (1968b) Social theory and social structure. New York: Free Press. [MDG, taDPP]Google Scholar
Merton, R. K. (1973) The sociology of science. Chicago: University of Chicago Press. [DLE]Google Scholar
Michie, D. (1978) Peer review and the bureaucracy. Times Higher Education Supplement 08 4, p. 11. [CM]Google Scholar
Mitroff, I. I. (1974) The subjective side of science. Amsterdam: Elsevier. [1IM]Google Scholar
Mitroff, I. I. & Chubin, D. E. (1979) Peer review at NSF: A dialectical policy analysis. Social Studies of Science 9:199–232. [DEC, CM]CrossRefGoogle Scholar
Moore, M. (1978) Discrimination or favoritism? Sex bias in book reviews. American Psychologist 33:936–38. [RO]CrossRefGoogle Scholar
Moravcsik, M. J. (1980) How to grow science. New York: Universe Books. [Ed.]Google Scholar
Moyal, A. (1980) The Australian Academy of Science: The anatomy of a scientific elite: parts 1 and 2. Search (ANZAAS) 11:231–39, 281–88. [CM]Google Scholar
Neisser, U. (1976) Cognition and reality: Principles and implications of cognitive psychology. San Francisco: Freeman. [taDPP]Google Scholar
NIH Grants Peer Review Study Team (1978) Grants peer review. Opinions on the NIH grants peer review system. Report to the Director, NIH. [RTL]Google Scholar
NIH Study Committee, Dean E. Wooldridge, chairman. (1965) Biomedical science and its administration: A study of the National Institutes of Health. Washington, D.C.: The White House. [RTL]Google Scholar
Nisbett, R. E. & Wilson, T. D. (1977) The halo effect: Evidence for unconscious alterations of judgments. Journal of Personality and Social Psychology 35:250–56. [RO]CrossRefGoogle Scholar
Oromaner, M. (1977) Professional age and the reception of sociological publications: A test of the Zuckerman-Merton hypothesis. Social Studies of Science 7:381–88. [taDPP]CrossRefGoogle Scholar
Orr, R. & Kassab, J. (1965) Peer group judgment on scientific merit: Editorial refereeing. Presented to the Congress of the International Federation for Documentation, Washington, D.C. [BCG]Google Scholar
Over, R. (1981) Representation of women on the editorial boards of psychology journals. American Psychologist 36:885–91. [RO]CrossRefGoogle Scholar
Patterson, E. H. (1969) Evaluation of manuscripts submitted for publication. American Psychologist 24:73. [DVC]CrossRefGoogle Scholar
Peters, D. P., & Ceci, S. J. (1980) A manuscript masquerade. How well does the review process work? Sciences 20:16–19. [JJB]CrossRefGoogle Scholar
Pfeffer, J.; Leong, A. & Strehl, K. (1977) Paradigm development and particularism: Journal publication in three scientific disciplines. Social Forces 55:938–51. [JMB]CrossRefGoogle Scholar
Pheterson, G. I.; Kiesler, S. B. & Goldberg, P. A. (1971) Evaluation of the performance of women as a function of their sex, achievement, and personal history. Journal of Personality and Social Psychology 19:114–18. [RO]CrossRefGoogle Scholar
Price, D. (1970) Citation measures of hard science, soft science, technology and nonscience. In: Communications among scientists and engineers, ed. Nelson, C. E. & Pollack, D.. Lexington, Mass.: D. C. Heath and Company. [BCG]Google Scholar
Ravetz, J. R. (1981) Avoiding fraud (correspondence). Nature 291:7. [CM]CrossRefGoogle Scholar
Revusky, S. (1977) Interference with progress by the scientific establishment: Examples from flavor aversion learning. In: Food aversion learning, ed. Milgram, N. W., Knames, L. & Alloway, T. M.. London: Plenum. [JH, taDPP, CJT]Google Scholar
Robertson, P. (1976) Towards open refereeing. New Scientist 71:410. [RAG]Google Scholar
Roose, K. D. & Anderson, C. J. (1970) A rating of graduate programs. Washington, D.C.: American Council on Education. [DP, taDPP]Google Scholar
Rosenblatt, A. & Kirk, S. A. (1980) Recognition of authors in blind review of manuscripts. Journal of Social Service Research 3:383–94. [tarDPP]CrossRefGoogle Scholar
Rosenthal, R. (1966) Experimenter effects in behavioral research. New York: Appleton-Century-Crofts; rev. ed., New York: Irvington, 1976. [taDPP, RR]Google Scholar
Rosenthal, R. & Rubin, D. B. (1978) Interpersonal expectancy effects: The first 345 studies. Behavioral and Brain Sciences 3:377–86. [tarDPP]CrossRefGoogle Scholar
Ross, C. (1979) Rejected. New West 4:39–41. [JSA, CR]Google Scholar
Ross, C. (1980) Editors choice: None of the above. Las Angeles Times, 02 17, p. 3. [JSA, CR]Google Scholar
Ross, P. F. (1981) The sciences' self-management: Manuscript refereeing, peer review, and goals in science. Unpublished manuscript. [taDPP]Google Scholar
Rowney, J. A. & Zenisek, T. J. (1980) Manuscript characteristics influencing reviewers' decisions. Canadian Psychology 21:17–21. [DP]CrossRefGoogle Scholar
Roy, R. (1981) An alternative funding mechanism. Science 211:1377. [DEC]CrossRefGoogle Scholar
Ruderfer, M. (1980) The fallacy of peer review – Judgment without science and a case history. Speculations in Science and Technology 3:533–62. [RAG, WMH, taDPP]Google Scholar
Rushton, J. P. & Roediger, H. L. (1978) An evaluation of 80 psychology journals based on the Science Citation Index. American Psychologist 33:520–23. [taDPP]CrossRefGoogle Scholar
Saracevic, T. (1975) Relevance: A review and framework for thinking on the notion in information science. Journal of the American Society for Information Science 26:321–13. [BCG]CrossRefGoogle Scholar
Sayre, A. (1975) Kosnlind Franklin and DNA. New York: Norton. [CM]Google Scholar
Scarr, S. & Weber, B. L. R. (1978) The reliability of reviews for the American Psychologist. American Psychologist 33:935. [tarDPP, GJW, JCW]CrossRefGoogle Scholar
Schaeffer, D. L. (1970) Do APA journals play professional favorites? American Psychologist 25:362–65. [JSA]CrossRefGoogle Scholar
Scott, W. A. (1974) Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology. American Psychologist 29:698–702. [DVC, taDPP, WAS, GJW]CrossRefGoogle Scholar
Scott, W. A. (1977) Methodological consensus among journal referees: A follow-up study of JPSP manuscripts. Unpublished manuscript. [WAS]Google Scholar
Shaw, R. & Bransford, J., eds. (1977) Perceiving, acting, and knowing: Toward an ecological psychology. Hillsdale, N.J.: Lawrence Erlbaum. [MJM]Google Scholar
Siegfried, J. J. (1970) A first lesson in econometrics. Jottrnal of Political Economy 78:1378–79. [JSA]CrossRefGoogle Scholar
Slovic, P. & Fischhoff, B. (1977) On the psychology of experimental surprises. Journal of Experimental Psychology: Human Perception and Performance 3:544–51. [JSA]Google Scholar
Smigel, E. D. & Ross, H. L. (1970) Factors in the editorial decision. American Sociologist 5:19–21. [DVC]Google Scholar
Snizek, W. E., Fuhrmam, E. R. & Wood, M. R. (1981) The effect of theory group association on the evaluative content of book reviews in sociology. American Sociologist 16:185–95. [OLE]Google Scholar
Social Science Citation Index. (1977) Guide and journal lists. Philadelphia: Institute for Scientific Information. [DLE]Google Scholar
Stnrk-Adamec, C. & Adamec, R. (in press) Breaking into the grant proposal market. International Journal of Women's Studies. [rDPP]Google Scholar
Stinchcombe, A. & Ofshe, R. (1969) On journal editing as a probabilistic process. American Sociologist 4:116–17. [SP, rDPP]Google Scholar
Stumpf, W. E. (1980) “Peer” review. Science 207:822–23. [taDPP]CrossRefGoogle ScholarPubMed
Swindel, R. F. & Perry, T. O. (1975) A previously unannounced form of the Gaussian distribution: The golden rule of arts and sciences. Journal of Irreproducible Results 21:8–9. [DVC]Google Scholar
Symposium. (1979) Reviews of Landsey's The scientific publication system in social science. Contemporary Sociology 8:814–24. [DEC]Google Scholar
Szasz, T. (1973) The second sin. London: Routledge and Kegan Paul. [JSA]Google Scholar
Tobach, E. (1980) “… that ye be judged.” In An evaluation of the peer review system in psychological research, chairman J. Demarest, open forum presented at the American Psychological Convention, Montreal. [taDPP]Google Scholar
Trafford, A. (1981) Behind the scandals in science labs. U.S. News and World Report, 03 2, p. 54. [JSA]Google Scholar
Tuckman, H. P. & Leaky, J. (1975) What is an article worth? Journal of Political Economy 83:951–67. [RC]CrossRefGoogle Scholar
Virgo, J. (1974) A statistical procedure for evaluating the importance of scientific papers. Ph.D. thesis, University of Chicago. [BCG]Google Scholar
Walster, G. W. & deary, T. A. (1970) A proposal for a new editorial policy in the social sciences. American Statistician 24:16–19. [taDPP]Google Scholar
Watkins, M. W. (1979) Chance and interrater agreement on manuscripts. American Psychologist 34:796–97. [RMP, tarDPP, JCW]CrossRefGoogle Scholar
Webb, W. B. (1979) Continuing education: Refereeing journal articles. Teaching Psychology 6:59–60. [taDPP]Google Scholar
Webster, E. C. (1964) Decision making in the employment interview. Montreal: Eagle. [JSA]Google Scholar
Weick, K. E. (in press) Systematic observational methods. In: The handbook of social psychology, ed. G. Lindzey & E. Aronson. 3d ed. [JDC]Google Scholar
Weimer, W. B. (1977) A conceptual framework for cognitive psychology: Motor theories of the mind. In: Perceiving, acting, and knowing: Toward an cological psychology, eds. Shaw, R. & Bransford, J., pp. 267–311. Hillsdale, N.J.: Lawrence Erlbaum. [MJM]Google Scholar
Weimer, W. B. (1979) Notes on the methodology of scientific research. Hillsdale, N.J.: Lawrence Erlbaum. [MJM]Google Scholar
Weimer, W. B. & Palermo, D. S., eds. (1981) Cognition and the symbolic processes. Vol. 2. Hillsdale, N.J.: Lawrence Erlbaum. [MJM]Google Scholar
Weiss, R. J. (1980). The use and abuse of deception. American Journal of Public Health 70:1097–98. [JLF, rDPP]CrossRefGoogle ScholarPubMed
White, M. J. & White, K. G. (1977) Citation analysis of psychology journals. American Psychologist 32:301–5. [tarDPP, MJW]CrossRefGoogle Scholar
Winer, B. J. (1971). Statistical principles in experimental design. 2d ed.New York: McGraw-Hill. [JLF]Google Scholar
Wolff, W. M. (1973) Publication problems in psychology and an explicit evaluation schema for manuscripts. American Psychologist 28:257–61. [taDPP]CrossRefGoogle Scholar
Wolin, L. (1962) Responsibility for raw data. American Psychologist 17:657–58. [JSA]CrossRefGoogle Scholar
Wright, R. D. (1970) Truth and its keepers. New Scientist 45:402–4. [LD]Google Scholar
Yalow, R. S. (1978) Radioimmunoassay: A probe for the fine structure of biology systems. In: Les prix nobel en 1977, pp. 243–64. Nobel Foundation. Stockholm: Almqvist & Wiksell. [RSY]Google Scholar
Yoels, W. (1974) The structure of scientific fields and the allocation of editorship on scientific journals: Some observations on the politics of knowledge. Sociological Quarterly 15:264–76. [JMB]CrossRefGoogle Scholar
Yotopoulos, P. A. (1961) Institutional affiliation of the contributors to three professional journals. American Economic Review 51:665–70. [taDPP]Google Scholar
Ziman, J. (1968) Public knowledge: The social dimension of science. Cambridge: Cambridge University Press. [MJM, CM, BM]Google Scholar
Ziman, J. (1976) The force of knowledge: The scientific dimension of society. Cambridge: Cambridge University Press. [AMC]Google Scholar
Zinberg, D. S. (1976) Education through science: The early stages of career development in chemistry. Social Studies of Science 6:215–46. [CM]CrossRefGoogle Scholar
Zuckerman, H. (1970) Stratification in American science. Sociological Inquiry 40:235–57. [taDPP]CrossRefGoogle Scholar
Zuckerman, H. & Merton, R. (1973) Patterns of evaluation in science: Institutionalization, structure and functions of the referee system. In: The scoiology of science, ed. Storer, N.. Chicago: University of Chicago Press. [BCG, taDPP, SP]Google Scholar