Hostname: page-component-745bb68f8f-l4dxg Total loading time: 0 Render date: 2025-01-08T11:59:41.978Z Has data issue: false hasContentIssue false
  • English
  • Français

On the Number of Significant Findings to be Expected by Chance

Published online by Cambridge University Press:  01 January 2025

Jack Block
Jack Block*
Affiliation:
University of California, Berkeley
Get access Rights & Permissions [Opens in a new window]

Abstract

When multiple significance tests are computed, a certain number of “significant” findings will emerge simply because of chance fluctuations. In the present paper, some factors affecting the number of nominally significant results are elaborated and a general method is suggested which permits unbiased inference as to the significance of a set of findings, as a set. The method advocated employs a high speed computer to generate empirically a sampling distribution tailormade to a particular data matrix. The method is illustrated in the case of dichotomous response to inventory items, where it is found that the statistical model still often used as a basis for estimation is overly conservative. Some problems in the application of the method are discussed.

Type
Original Paper
Information
Psychometrika , Volume 25 , Issue 4 , December 1960 , pp. 369 - 380
Copyright
Copyright © 1960 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

*

This investigation, part of a larger research project, was supported primarily by research grant M-1078 from the National Institute of Mental Health of the United States Public Health Service. Gratitude is also extended to the National Science Foundation for making available research time on the IBM Model 701 computer. The present paper has benefited greatly from the comments on an earlier version offered by John Tukey and an editor of Psychometrika.

References

Arthur, A. O. Random digit generation. Computing News, 1956, 4, 85–85.Google Scholar
Brozek, J. and Tiede, K. Reliable and questionable significance in a series of statistical tests. Psychol. Bull., 1952, 49, 339–341.CrossRefGoogle Scholar
Johnson, D. L. Generating and testing pseudo-random numbers on the IBM Type 701. Mathematical tables and other aids to computation, 1956, 10, 8–13.CrossRefGoogle Scholar
Jones, L. V. and Fiske, D. W. Models for testing the significance of combined results. Psychol. Bull., 1953, 50, 375–382.CrossRefGoogle ScholarPubMed
Latscha, R. Tests of significance in a 2 × 2 contingency table: extension of Finney's table. Biometrika, 1953, 40, 74–86.Google Scholar
Meyer, H. A. Symposium on Monte Carlo methods, New York: Wiley, 1956.Google Scholar
Mosteller, F. and Bush, R. R. Selected quantitative techniques. In Lindzey, G. (Eds.), Handbook of social psychology. Cambridge, Mass.: Addison-Wesley, 1954, 289–334.Google Scholar
Sakoda, J. M., Cohen, B. H., and Beall, G. Test of significance for a series of statistical tests. Psychol. Bull., 1954, 51, 172–175.CrossRefGoogle ScholarPubMed
Tukey, J. W. Comparing two small samples on many items. Memo. Rep. 54, Statist. Res. Group, Princeton Univ., 1954.Google Scholar
Wilkinson, B. A statistical consideration in psychological research. Psychol. Bull., 1951, 48, 156–158.CrossRefGoogle ScholarPubMed