Hostname: page-component-745bb68f8f-grxwn Total loading time: 0 Render date: 2025-01-08T12:05:48.515Z Has data issue: false hasContentIssue false
  • English
  • Français

Reliability as a Function of the Number of Item Options Derived from the “Knowledge or Random Guessing” Model

Published online by Cambridge University Press:  01 January 2025

Robert G. MacCann
Robert G. MacCann*
Affiliation:
Measurement and Research Services, Board of Studies NSW, Sydney, Australia
*
Requests for reprints should be addressed to: Dr. Robert MacCann, Head, Measurement & Research Services, Board of Studies NSW, GPO Box 5300, Sydney, AUSTRALIA 2001 E-mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

For (0, 1) scored multiple-choice tests, a formula giving test reliability as a function of the number of item options is derived, assuming the “knowledge or random guessing model,” the parallelism of the new and old tests (apart from the guessing probability), and the assumptions of classical test theory. It is shown that the formula is a more general case of an equation by Lord, and reduces to Lord's equation if the items are effectively parallel. Further, the formula is shown to be closely related to another formula derived from Lord's randomly parallel tests model.

Keywords

guessing (tests)test reliabilitydistractorstest lengtherror of measurement
Type
Notes And Comments
Information
Psychometrika , Volume 69 , Issue 1 , March 2004 , pp. 147 - 157
Copyright
Copyright © 2004 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Carroll, J. B. (1945). The effect of difficulty and chance success on correlations between items or between tests. Psychometrika, X, 1–19CrossRefGoogle Scholar
Ebel, R. L. (1969). Expected reliability as a function of choices per item. Educational and Psychological Measurement, 29, 565–570CrossRefGoogle Scholar
Feldt, L.S., & Brennan, R.L. (1989). Reliability. In Linn, R.L. (Eds.), Educational Measurement 3rd ed., (pp. 105–147). New York: American Council on Education; MacmillanGoogle Scholar
Grier, J. B. (1975). The number of alternatives for optimum test reliability. Journal of Educational Measurement, 12, 109–113CrossRefGoogle Scholar
Horst, P. (1954). The estimation of immediate retest reliability. Educational and Psychological Measurement, 14, 705–708CrossRefGoogle Scholar
Komaroff, E. (1997). Effect of simultaneous violations of essential tau-equivalence and uncorrelated error on Coefficient alpha. Applied Psychological Measurement, 21, 337–348CrossRefGoogle Scholar
Lord, F. M. (1944). Reliability of multiple choice tests as a function of choices per item. Journal of Educational Psychology, 35, 175–180CrossRefGoogle Scholar
Lord, F. M. (1957). Do tests of the same length have the same standard error of measurement?. Educational and Psychological Measurement, 17, 510–521CrossRefGoogle Scholar
Lord, F. M. (1959). Tests of the same length do have the same standard error of measurement. Educational and Psychological Measurement, 19, 233–239CrossRefGoogle Scholar
Lord, F. M. (1977). Optimal number of choices per item—a comparison of four approaches. Journal of Educational Measurement, 14, 33–38CrossRefGoogle Scholar
Lord, F. M., & Novick, M. R. (1968). Statistical Theories of Mental Test Scores. Reading, MA: Addison-WesleyGoogle Scholar
Mattson, D. (1965). The effects of guessing on the standard error of measurement and the reliability of test scores. Educational and Psychological Measurement, 25, 727–730CrossRefGoogle Scholar
Novick, M.R., & Lewis, C. (1967). Coefficient alpha and the reliability of composite measurements. Psychometrika, 32, 1–13CrossRefGoogle ScholarPubMed
Raykov, T. (2001). Estimation of congeneric scale reliability using covariance structure analysis with nonlinear constraints. British Journal of Mathematical and Statistical Psychology, 54, 315–323CrossRefGoogle ScholarPubMed
Trevisan, M.S., Sax, G., & Michael, W.B. (1994). Estimating the optimum number of options using an incremental option paradigm. Educational and Psychological Measurement, 54, 86–91CrossRefGoogle Scholar
Zimmerman, D.W. (1975). Probability spaces, Hilbert spaces, and the axioms of test theory. Psychometrika, 40, 395–412CrossRefGoogle Scholar
Zimmerman, D.W. (1976). Test theory with minimal assumptions. Educational and Psychological Measurement, 36, 85–96CrossRefGoogle Scholar
Zimmerman, D.W., Zumbo, B.D., & Lalonde, C. (1993). Coefficient alpha as an estimate of test reliability under violation of two assumptions. Educational and Psychological Measurement, 53, 33–49CrossRefGoogle Scholar