Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-6dbcb7884d-fw45b Total loading time: 0 Render date: 2025-02-14T06:48:38.293Z Has data issue: false hasContentIssue false

9 - Reliability

from Part II - Important Methodological Considerations

Published online by Cambridge University Press:  12 December 2024

By
Tenko Raykov
Edited by
John E. Edlund  and
Austin Lee Nichols
John E. Edlund
Affiliation:
Rochester Institute of Technology, New York
Austin Lee Nichols
Affiliation:
Central European University, Vienna
Get access

Summary

This chapter is concerned with reliability as a key indicator of measurement quality in behavioral and social science research. It commences with a discussion of the basics and a definition of the reliability coefficient. The following section deals with the meaning, interpretation, and utility of the reliability concept. Subsequently, the focus is on the evaluation of reliability as well as its discrepancy from the popular coefficient alpha that has been widely used for a number of decades as an index related to reliability. The large-sample behavior of the alpha and scale reliability estimates is then discussed, as is the relationship between the reliability coefficient and that of standardized reliability. The conclusion points out the limitations of the procedures for reliability evaluation discussed in the chapter.

Type
Chapter
Information
The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
Volume 2: Performing Research
 , pp. 189 - 210
Publisher: Cambridge University Press
Print publication year: 2024

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Agresti, A., & Finlay, B. (2008). Statistical Methods for the Social Sciences. CRC Press.Google Scholar
Arnold, S. F. (1990). Mathematical Statistics. Prentice Hall.Google Scholar
Bartholomew, D. J. (1996). The Statistical Approach to Social Measurement. Academic Press.Google Scholar
Bevington, P. R., & Robinson, D. K. (2003). Data Reduction and Error Analysis for the Physical Sciences . McGraw-Hill.Google Scholar
Bollen, K. A. (1980). Issues in the comparative measurement of political democracy. American Sociological Review, 45, 370–390.CrossRefGoogle Scholar
Bollen, K. A. (1989). Structural Equations with Latent Variables. Wiley.CrossRefGoogle Scholar
Casella, G., & Berger, J. O. (2002). Statistical Inference. Wadsworth.Google Scholar
Crocker, L., & Algina, J. (1986). Introduction to Classical and Modern Test Theory. Harcourt College Publishers.Google Scholar
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.CrossRefGoogle Scholar
DasGupta, A. (2008). Asymptotic Theory of Statistics and Probability. Springer.Google Scholar
Efron, B., & Tibshiriani, R. J. (1993). An Introduction to the Bootstrap. Chapman Hall/CRC.CrossRefGoogle Scholar
Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for Psychologists. Lawrence Erlbaum Associates.Google Scholar
Hu, L.-T., Bentler, P. M., & Kano, Y. (1992). Can test statistics in covariance structure analysis be trusted? Psychological Bulletin, 112(2), 351–362.CrossRefGoogle ScholarPubMed
Jöreskog, K. G. (1971). Statistical analysis of sets of congeneric tests. Psychometrika, 36, 109–133.CrossRefGoogle Scholar
Judd, C. M., McClelland, G. H., & Ryan, C. S. (2017). Data analysis: A Model Comparison Approach to Regression, ANOVA, and Beyond, 3rd ed. Routledge.CrossRefGoogle Scholar
Kelley, T. L. (1927). Interpretation of Educational Measurement. World Book Company.Google Scholar
Lord, F. M., & Novick, M. (1968). Statistical Theories of Mental Test Scores. Wesley.Google Scholar
Maxwell, A. E. (1967). The effect of correlated errors on estimates of reliability coefficients. Education and Psychological Measurement, 28, 803–811.CrossRefGoogle Scholar
McDonald, R. P. (1999). Test Theory: A Unified Treatment. Lawrence Erlbaum Associates.Google Scholar
McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23, 412–433.CrossRefGoogle Scholar
Novick, M. R., & Lewis, C. (1967). Coefficient alpha and the reliability of composite measurements. Psychometrika, 32, 1–13.CrossRefGoogle ScholarPubMed
Rao, C. R. (1973). Linear Statistical Inference and Its Applications. Wiley.CrossRefGoogle Scholar
Raykov, T. (1997). Scale reliability, Cronbach’s coefficient alpha, and violations of essential tau-equivalence for fixed congeneric components. Multivariate Behavioral Research, 32, 329–354.CrossRefGoogle ScholarPubMed
Raykov, T. (2001a). Bias of coefficient alpha for congeneric measures with correlated errors. Applied Psychological Measurement, 25, 69–76.CrossRefGoogle Scholar
Raykov, T. (2001b). Estimation of congeneric scale reliability via covariance structure models with nonlinear constraints. British Journal of Mathematical and Statistical Psychology, 54, 315–323.CrossRefGoogle Scholar
Raykov, T. (2007). Reliability if deleted, not “alpha if deleted”: Evaluation of scale reliability following component deletion. British Journal of Mathematical and Statistical Psychology, 60, 201–216.CrossRefGoogle Scholar
Raykov, T. (2008). “Alpha if item deleted”: A note on loss of criterion validity in scale development if maximising coefficient alpha. British Journal of Mathematical and Statistical Psychology, 61, 275–285.CrossRefGoogle Scholar
Raykov, T. (2009). Evaluation of scale reliability for unidimensional measures using latent variable modeling. Measurement and Evaluation in Counseling and Development, 42, 222–232.CrossRefGoogle Scholar
Raykov, T. (2012). Scale development using structural equation modeling. In Hoyle, R. (ed.), Handbook of Structural Equation Modeling (pp. 472–492). Guilford Press.Google Scholar
Raykov, T. (2019a). Strong consistency of reliability estimators for multiple-component measuring instruments. Structural Equation Modeling, 26, 750–756.CrossRefGoogle Scholar
Raykov, T. (2019b). Strong convergence of the coefficient alpha estimator for reliability of multiple-component measuring instruments. Structural Equation Modeling, 26, 430–436.CrossRefGoogle Scholar
Raykov, T. (2023). Psychometric scale evaluation using structural equation and latent variable modeling. In Hoyle, R. (ed.), Handbook of Structural Equation Modeling, 2nd ed. Guilford Press.Google Scholar
Raykov, T., Anthony, J. C., & Menold, N. (2023). On the importance of coefficient alpha for measurement research: Loading equality is not necessary for alpha’s utility as a scale reliability index. Educational and Psychological Measurement, 83(4), 766–781.CrossRefGoogle Scholar
Raykov, T., Dimitrov, D. M., & Asparouhov, T. (2010). Evaluation of scale reliability with binary measures using latent variable modeling. Structural Equation Modeling, 17, 122–132.CrossRefGoogle Scholar
Raykov, T., Doebler, P., & Marcoulides, G. A. (2022). Applications of Bayesian confirmatory factor analysis in behavioral measurement: Strong convergence of a Bayesian parameter estimator. Measurement: Interdisciplinary Research and Perspectives, 20(4), 215–227.Google Scholar
Raykov, T., & Marcoulides, G. A. (2004). Using the delta method for approximate interval estimation of parametric functions in covariance structure models. Structural Equation Modeling, 11, 659–675.CrossRefGoogle Scholar
Raykov, T., & Marcoulides, G. A. (2006). A first Course in Structural Equation Modeling. Lawrence Erlbaum Associates.Google Scholar
Raykov, T., & Marcoulides, G. A. (2011). Introduction to Psychometric Theory. Routledge.CrossRefGoogle Scholar
Raykov, T., & Marcoulides, G. A. (2015a). A direct latent variable modeling-based procedure for evaluation of coefficient alpha. Educational and Psychological Measurement, 75, 146–156.CrossRefGoogle Scholar
Raykov, T., & Marcoulides, G. A. (2015b). Scale reliability evaluation in heterogeneous populations. Educational and Psychological Measurement, 75, 875–892.CrossRefGoogle ScholarPubMed
Raykov, T., & Marcoulides, G. A. (2016a). Scale reliability evaluation under multiple assumption violations. Structural Equation Modeling, 23, 302–313.CrossRefGoogle Scholar
Raykov, T., & Marcoulides, G. A. (2016b). On examining specificity in latent construct indicators. Structural Equation Modeling, 23, 845–855.CrossRefGoogle Scholar
Raykov, T., & Marcoulides, G. A. (2018). A Course in Item Response Theory and Modeling with Stata. Stata Press.Google Scholar
Raykov, T., & Marcoulides, G. A. (2019). Thanks, coefficient alpha – we still need you! Educational and Psychological Measurement, 79, 200–210.CrossRefGoogle ScholarPubMed
Raykov, T., & Marcoulides, G. A. (2021). On the pitfalls of estimating and using standardized reliability coefficients. Educational and Psychological Measurement, 791–810.CrossRefGoogle Scholar
Raykov, T., & Marcoulides, G. A. (2023). Evaluating the discrepancy between scale reliability and Cronbach’s coefficient alpha using latent variable modeling. Measurement: Interdisciplinary Research and Perspectives, 21(1), 29–37.Google Scholar
Raykov, T., Marcoulides, G. A., & Chang, C. (2016). Studying population heterogeneity in finite mixture settings using latent variable modeling. Structural Equation Modeling, 23, 726–730.CrossRefGoogle Scholar
Raykov, T., Marcoulides, G. A., & Li, T. (2016). Measurement instrument validity evaluation in finite mixtures. Educational and Psychological Measurement, 76, 1026–1044.CrossRefGoogle Scholar
Raykov, T., Marcoulides, G. A., & Patelis, T. (2015). The importance of the assumption of uncorrelated errors in psychometric theory. Educational and Psychological Measurement, 75, 634–647.CrossRefGoogle ScholarPubMed
Raykov, T., & Shrout, P. E. (2002). Reliability of scales with general structure: Point and interval estimation using a structural equation modeling approach. Structural Equation Modeling, 9, 195–212.CrossRefGoogle Scholar
Raykov, T., West, B. T., & Traynor, A. (2015). Evaluation of coefficient alpha for multiple component measuring instruments in complex sample designs. Structural Equation Modeling, 22, 429–438.CrossRefGoogle Scholar
Rhemtulla, M., Brosseau-Liard, P. E., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354–373.CrossRefGoogle ScholarPubMed
Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107–120.CrossRefGoogle ScholarPubMed
Spearman, C. (1904). “General intelligence,” objectively determined and measured. American Journal of Psychology, 15, 201–292.CrossRefGoogle Scholar
Yang, Y., & Green, S. B. (2009). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 75, 155–167.Google Scholar
Zimmerman, D. W. (1972). Test reliability and the Kuder-Richardson formulas: Derivation from probability theory. Educational and Psychological Measurement, 32, 939–954.CrossRefGoogle Scholar
Zimmerman, D. W. (1975). Probability spaces, Hilbert spaces, and the axioms of test theory. Psychometrika, 40, 395–412.CrossRefGoogle Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

  • Reliability
  • Edited by John E. Edlund, Rochester Institute of Technology, New York, Austin Lee Nichols, Central European University, Vienna
  • Book: The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
  • Online publication: 12 December 2024
  • Chapter DOI: https://doi.org/10.1017/9781009000796.010
Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

  • Reliability
  • Edited by John E. Edlund, Rochester Institute of Technology, New York, Austin Lee Nichols, Central European University, Vienna
  • Book: The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
  • Online publication: 12 December 2024
  • Chapter DOI: https://doi.org/10.1017/9781009000796.010
Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

  • Reliability
  • Edited by John E. Edlund, Rochester Institute of Technology, New York, Austin Lee Nichols, Central European University, Vienna
  • Book: The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
  • Online publication: 12 December 2024
  • Chapter DOI: https://doi.org/10.1017/9781009000796.010
Available formats
×