Hostname: page-component-cd9895bd7-jkksz Total loading time: 0 Render date: 2025-01-06T03:50:39.420Z Has data issue: false hasContentIssue false
  • English
  • Français

Scalable Bayesian Approach for the Dina Q-Matrix Estimation Combining Stochastic Optimization and Variational Inference

Published online by Cambridge University Press:  01 January 2025

Motonori Oka Open the ORCID record for Motonori Oka [Opens in a new window]  and
Kensuke Okada Open the ORCID record for Kensuke Okada [Opens in a new window]
Motonori Oka*
Affiliation:
The University of Tokyo
Kensuke Okada
Affiliation:
The University of Tokyo
*
Correspondence should be made to Motonori Oka, Graduate School of Education, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan. Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Diagnostic classification models offer statistical tools to inspect the fined-grained attribute of respondents’ strengths and weaknesses. However, the diagnosis accuracy deteriorates when misspecification occurs in the predefined item–attribute relationship, which is encoded into a Q-matrix. To prevent such misspecification, methodologists have recently developed several Bayesian Q-matrix estimation methods for greater estimation flexibility. However, these methods become infeasible in the case of large-scale assessments with a large number of attributes and items. In this study, we focused on the deterministic inputs, noisy “and” gate (DINA) model and proposed a new framework for the Q-matrix estimation to find the Q-matrix with the maximum marginal likelihood. Based on this framework, we developed a scalable estimation algorithm for the DINA Q-matrix by constructing an iteration algorithm that utilizes stochastic optimization and variational inference. The simulation and empirical studies reveal that the proposed method achieves high-speed computation, good accuracy, and robustness to potential misspecifications, such as initial value choices and hyperparameter settings. Thus, the proposed method can be a useful tool for estimating a Q-matrix in large-scale settings.

Keywords

Q-matrix estimationstochastic optimizationvariational inferencediagnostic classification modelsdeterministic inputs noisy “and” gate (DINA) model
Type
Theory and Methods
Information
Psychometrika , Volume 88 , Issue 1 , March 2023 , pp. 302 - 331
Copyright
Copyright © 2022 The Author(s) under exclusive licence to The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Supplementary Information The online version contains supplementary material available at https://doi.org/10.1007/s11336-022-09884-4.

References

Beal, M. J. (2003). Variational algorithms for approximate Bayesian inference [Unpublished doctoral dissertation, University College London]. Retrieved from https://www.cse.buffalo.edu/faculty/mbeal/thesis/.Google Scholar
Bezanson, J., Edelman, A., Karpinski, S., Shah, V. B., (2017). Julia: A fresh approach to numerical computing SIAM Review 59(1) 65–98 10.1137/141000671CrossRefGoogle Scholar
Bishop, C. M., Pattern recognition and machine learning. Information science and statistics New York, USA SpringerGoogle Scholar
Blei, D. M., Kucukelbir, A., McAuliffe, J. D., (2006). Variational inference: A review for statisticians Journal of the American Statistical Association (2017). 112(518) 859–877 10.1080/01621459.2017.1285773CrossRefGoogle Scholar
Blei, D. M., Andrew, Y. N., Michael, I. J., (2003). Latent Dirichlet allocation Journal of Machine Learning Research 3 993–1022Google Scholar
Cai, L. (2010a). High-dimensional exploratory item factor analysis by a Metropolis–Hastings Robbins–Monro algorithm. Psychometrika, 75(1), 33–57. https://doi.org/10.1007/s11336-009-9136-x.CrossRefGoogle Scholar
Cai, L. (2010b). Metropolis–Hastings Robbins–Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35(3), 307–335. https://doi.org/10.3102/1076998609353115CrossRefGoogle Scholar
Camilli, G., Geis, E., (2019). Stochastic approximation EM for large-scale exploratory IRT factor analysis Statistics in Medicine 38(21) 3997–4012 10.1002/sim.8217 31267550CrossRefGoogle ScholarPubMed
Chalmers, R. P., Flora, D. B., (2014). Maximum-likelihood estimation of noncompensatory IRT models with the MH-RM algorithm Applied Psychological Measurement 38(5) 339–358 10.1177/0146621614520958CrossRefGoogle Scholar
Chen, J., de la Torre, J., (2014). A procedure for diagnostically modeling extant large-scale assessment data: The case of the programme for international student assessment in reading Psychology 05(18) 1967–1978 10.4236/psych.2014.518200CrossRefGoogle Scholar
Chen, Y., Culpepper, S. A., Chen, Y., Douglas, J., (2018). Bayesian estimation of the DINA Q matrix Psychometrika 83(1) 89–108 10.1007/s11336-017-9579-4 28861685CrossRefGoogle ScholarPubMed
Chen, Y., Culpepper, S., Liang, F., (2020). A sparse latent class model for cognitive diagnosis Psychometrika 85(1) 121–153 10.1007/s11336-019-09693-2 31927684CrossRefGoogle ScholarPubMed
Chen, Y., Liu, J., Xu, G., Ying, Z., (2015). Statistical analysis of Q-matrix based diagnostic classification models Journal of the American Statistical Association 110(510) 850–866 10.1080/01621459.2014.934827 26294801CrossRefGoogle Scholar
Chiu, C-Y Douglas, J. A., Li, X., (2009). Cluster analysis for cognitive diagnosis: Theory and applications Psychometrika 74(4) 633–665 10.1007/s11336-009-9125-0CrossRefGoogle Scholar
Cho, A. E., Wang, C., Zhang, X., Xu, G., (2021). Gaussian variational estimation for multidimensional item response theory British Journal of Mathematical and Statistical Psychology 74 52–85 10.1111/bmsp.12219 33064318CrossRefGoogle ScholarPubMed
Chung, M., (2019). A Gibbs sampling algorithm that estimates the Q-matrix for the DINA model Journal of Mathematical Psychology 93 10.1016/j.jmp.2019.07.002CrossRefGoogle Scholar
Culpepper, S. A. (2019a). An exploratory diagnostic model for ordinal responses with binary attributes: Identifiability and estimation. Psychometrika, 84(4), 921–940. https://doi.org/10.1007/s11336-019-09683-4CrossRefGoogle Scholar
Culpepper, S. A. (2019b). Estimating the cognitive diagnosis Q matrix with expert knowledge: Application to the fraction-subtraction dataset. Psychometrika, 84(2), 333–357. https://doi.org/10.1007/s11336-018-9643-8CrossRefGoogle Scholar
Culpepper, S. A., Chen, Y., (2019). Development and application of an exploratory reduced reparameterized unified model Journal of Educational and Behavioral Statistics 44(1) 3–24 10.3102/1076998618791306CrossRefGoogle Scholar
de la Torre, J., (2011). The generalized DINA model framework Psychometrika 76(2) 179–199 10.1007/s11336-011-9207-7CrossRefGoogle Scholar
de la Torre, J., Douglas, J. A., (2004). Higher-order latent trait models for cognitive diagnosis Psychometrika 69(3) 333–353 10.1007/BF02295640CrossRefGoogle Scholar
DeCarlo, L. T., (2012). Recognizing uncertainty in the Q-matrix via a Bayesian extension of the DINA model Applied Psychological Measurement 36(6) 447–468 10.1177/0146621612449069CrossRefGoogle Scholar
Delyon, B., Lavielle, M., & Moulines, E. (1999). Convergence of a stochastic approximation version of the EM algorithm. The Annals of Statistics,. https://doi.org/10.1214/aos/1018031103.CrossRefGoogle Scholar
Dempster, A. P., Laird, N. M., Rubin, D. B., (1977). Maximum likelihood from incomplete data via the EM algorithm Journal of the Royal Statistical Society: Series B (Methodological) 39(1) 1–22 10.1111/j.2517-6161.1977.tb01600.xCrossRefGoogle Scholar
Fang, G., Liu, J., Ying, Z., (2019). On the identifiability of diagnostic classification models Psychometrika 84(1) 19–40 10.1007/s11336-018-09658-x 30673967CrossRefGoogle ScholarPubMed
Feng, Y., Habing, B. T., Huebner, A., (2014). Parameter estimation of the reduced RUM using the EM algorithm Applied Psychological Measurement 38(2) 137–150 10.1177/0146621613502704CrossRefGoogle Scholar
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., Rubin, D. B., Bayesian data analysis 3 Boca Raton CRC Press 10.1201/b16018Google Scholar
Gu, Y., Xu, G., (2013). The sufficient and necessary condition for the identifiability and estimability of the DINA model Psychometrika (2019). 84(2) 468–483 10.1007/s11336-018-9619-8 29728918CrossRefGoogle Scholar
Gu, Y., & Xu, G. (2020). Partial identifiability of restricted latent class models. The Annals of Statistics. https://doi.org/10.1214/19-AOS1878.CrossRefGoogle Scholar
Gu, Y., & Xu, G. (2021). A joint MLE approach to large-scale structured latent attribute analysis. Journal of the American Statistical Association. https://doi.org/10.1080/01621459.2021.1955689.CrossRefGoogle Scholar
Hartz, S., Roussos, L., (2008). The fusion model for skills diagnosis: Blending theory with practicality ETS Research Report Series 2008(2) 1–57 10.1002/j.2333-8504.2008.tb02157.xCrossRefGoogle Scholar
Henson, R. A., Templin, J. L., Willse, J. T., (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables Psychometrika 74(2) 191–210 10.1007/s11336-008-9089-5CrossRefGoogle Scholar
Hoffman, M. D., Blei, D. M., Wang, C., Paisley, J., (2013). Stochastic variational inference Journal of Machine Learning Research 14 1303–1347Google Scholar
Humphreys, K., Titterington, D. M., (2003). Variational approximations for categorical causal modeling with latent variables Psychometrika 68(3) 391–412 10.1007/BF02294734CrossRefGoogle Scholar
Jang, E. E., Kim, H., Vincett, M., Barron, C., & Russel, B. (2019). Improving IELTS reading test score interpretations and utilisation through cognitive diagnosis model-based skill profiling. IELTS Research Reports Online Series, No. 2. British Council, Cambridge Assessment English and IDP: IELTS Australia. Retrieved from https://www.ielts.org/research/research-reports/online-series-2019-2.Google Scholar
Jeon, M., Rijmen, F., Rabe-Hesketh, S., (2017). A variational maximization-maximization algorithm for generalized linear mixed models with crossed random effects Psychometrika 82(3) 693–716 10.1007/s11336-017-9555-zCrossRefGoogle Scholar
Junker, B. W., Sijtsma, K., (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory Applied Psychological Measurement 25(3) 258–272 10.1177/01466210122032064CrossRefGoogle Scholar
Kunina-Habenicht, O., Rupp, A. A., Wilhelm, O., (2012). The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models: Detection of model misspecification in DCMs Journal of Educational Measurement 49(1) 59–81 10.1111/j.1745-3984.2011.00160.xCrossRefGoogle Scholar
Lee, M. D., & Wagenmakers, E.-J. (2013). Bayesian cognitive modeling: A practical course. https://doi.org/10.1017/CBO9781139087759.CrossRefGoogle Scholar
Liu, C-W Andersson, B., Skrondal, A., (2020). A constrained Metropolis–Hastings Robbins–Monro algorithm for Q matrix estimation in DINA models Psychometrika 85(2) 322–357 10.1007/s11336-020-09707-4 32632838CrossRefGoogle ScholarPubMed
Liu, J., Xu, G., Ying, Z., (2012). Data-driven learning of Q-matrix Applied Psychological Measurement 36(7) 548–564 10.1177/0146621612456591 23926363CrossRefGoogle ScholarPubMed
Liu, J., Xu, G., Ying, Z., (2013). Theory of self-learning Q-matrix Bernoulli 19(5) 1790–1817 10.3150/12-BEJ430 24812537 4011940CrossRefGoogle ScholarPubMed
Mandt, S., Hoffman, M. D., Blei, D. M., (2017). Stochastic gradient descent as approximate Bayesian inference Journal of Machine Learning Research 18 1–35Google Scholar
Naesseth, A. C. (2018). Machine learning using approximate inference: Variational and sequential Monte Carlo methods [Unpublished doctoral dissertation, Linkoping University]. Retrieved from https://doi.org/10.3384/diss.diva-152647.CrossRefGoogle Scholar
Natesan, P., Nandakumar, R., Minka, T., & Rubright, J. D. (2016, September 27). Bayesian prior choice in IRT estimation using MCMC and variational Bayes. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2016.01422.CrossRefGoogle Scholar
Neu, G., & Rosasco, L. (2018). Iterate averaging regularization for stochastic gradient descent. In Proceedings of the 31st conference on learning theory, in PMLR (Vol. 75, pp. 3222–3242).Google Scholar
Polyak, B., (1990). New stochastic approximation type procedures Autom. i Telemekh. 7.98107(7) 98–107Google Scholar
Polyak, B. T., Juditsky, A. B., (1992). Acceleration of stochastic approximation by averaging SIAM Journal on Control and Optimization 30(4) 838–855 10.1137/0330046CrossRefGoogle Scholar
R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from https://www.R-project.org/.Google Scholar
Rijmen, F., Jeon, M., (2013). Fitting an item response theory model with random item effects across groups by a variational approximation method Annals of Operations Research 206(1) 647–662 10.1007/s10479-012-1181-7CrossRefGoogle Scholar
Robbins, M., Monro, S., (1951). Stochastic approximation method The Annals of Mathematical Statistics 22(3) 400–407 10.1214/aoms/1177729586CrossRefGoogle Scholar
Robitzsch, A., Kiefer, T., George, A. C., & Unlu, A. (2020). CDM: Cognitive diagnosis modeling. R package version 7.5-15. Retrieved from. https://cran.rproject.org/web/packages/CDM/index.htmlGoogle Scholar
Ross, S. M. (2013). Simulation (5th ed.). Academic Press.Google Scholar
Rupp, A. A., Templin, J., (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model Educational and Psychological Measurement 68(1) 78–96 10.1177/0013164407301545CrossRefGoogle Scholar
Ruppert, D. (1988). Efficient estimations from a slowly convergent Robbins-Monro process. Technical Report, Cornell University Operations Research and Industrial Engineering. Retrieved from https://hdl.handle.net/1813/8664.Google Scholar
Sessoms, J., Henson, R. A., (2018). Applications of diagnostic classification models: A literature review and critical commentary Measurement: Interdisciplinary Research and Perspectives 16(1) 1–17 10.1080/15366367.2018.1435104Google Scholar
Skaggs, G., Wilkins, J. L. M., Hein, S. F., (2016). Grain size and parameter recovery with TIMSS and the general diagnostic model International Journal of Testing 16(4) 310–330 10.1080/15305058.2016.1145683CrossRefGoogle Scholar
Spall, J. C. (2003). Introduction to stochastic search and optimization: Estimation, simulation, and control. Wiley-Interscience series in discrete mathematics and optimization. Wiley-Interscience.CrossRefGoogle Scholar
Su, Y-L Choi, K. M., Lee, W-C Choi, T., McAninch, M., (2013). Hierarchical cognitive diagnostic analysis for TIMSS 2003 mathematics Centre for Advanced Studies in Measurement and Assessment 35 1–71Google Scholar
Tatsuoka, K. K., (1983). Rule space: An approach for dealing with misconceptions based on item response theory Journal of Educational Measurement 20(4) 345–354 10.1111/j.1745-3984.1983.tb00212.xCrossRefGoogle Scholar
Tatsuoka, K. K., (2002). Data analytic methods for latent partially ordered classification models Journal of the Royal Statistical. Society Series C: Applied Statistics 51(3) 337–350 10.1111/1467-9876.00272CrossRefGoogle Scholar
Templin, J., Bradshaw, L., (2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies Psychometrika 79(2) 317–339 10.1007/s11336-013-9362-0 24478021CrossRefGoogle ScholarPubMed
Templin, J. L., Henson, R. A., (2006). Measurement of psychological disorders using cognitive diagnosis models Psychological Methods 11(3) 287–305 10.1037/1082-989X.11.3.287 16953706CrossRefGoogle ScholarPubMed
Templin, J., Hoffman, L., (2013). Obtaining diagnostic classification model estimates using Mplus Educational Measurement: Issues and Practice 32(2) 37–50 10.1111/emip.12010CrossRefGoogle Scholar
von Davier, M., (2008). A general diagnostic model applied to language testing data British Journal of Mathematical and Statistical Psychology 61(2) 287–307 10.1348/000711007X193957CrossRefGoogle ScholarPubMed
von Davier, M., Sinharay, S., (2010). Stochastic approximation methods for latent regression item response models Journal of Educational and Behavioral Statistics 35(2) 174–193 10.3102/1076998609346970CrossRefGoogle Scholar
Watanabe, S., (2010). Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory Journal of Machine Learning Research 11 3571–3594Google Scholar
Watanabe, S., (2013). A widely applicable Bayesian information criterion Journal of Machine Learning Research 14 867–897Google Scholar
Xu, G., (2017). Identifiability of restricted latent class models with binary responses The Annals of Statistics 45(2) 675–707 10.1214/16-AOS1464CrossRefGoogle Scholar
Xu, G., Shang, Z., (2018). Identifying latent structures in restricted latent class models Journal of the American Statistical Association 113(523) 1284–1295 10.1080/01621459.2017.1340889CrossRefGoogle Scholar
Xu, G., Zhang, S., (2016). Identifiability of diagnostic classification models Psychometrika 81(3) 625–649 10.1007/s11336-015-9471-z 26155755CrossRefGoogle ScholarPubMed
Yamaguchi, K., (2020). Variational Bayesian inference for the multiple-choice DINA model Behaviormetrika 47(1) 159–187 10.1007/s41237-020-00104-wCrossRefGoogle Scholar
Yamaguchi, K., Okada, K., (2018). Comparison among cognitive diagnostic models for the TIMSS 2007 fourth grade mathematics assessment PLOS ONE 13(2) 10.1371/journal.pone.0188691 29394257 5796692CrossRefGoogle ScholarPubMed
Yamaguchi, K., Okada, K., (2020). Variational Bayes inference for the DINA model Journal of Educational and Behavioral Statistics 45(5) 569–597 10.3102/1076998620911934CrossRefGoogle Scholar
Zhang, C., Butepage, J., Kjellstrom, H., Mandt, S., (2019). Advances in variational inference IEEE Transactions on Pattern Analysis and Machine Intelligence 41(8) 2008–2026 10.1109/TPAMI.2018.2889774 30596568CrossRefGoogle ScholarPubMed
Yamaguchi, K., Okada, K., (2021). Variational Bayes inference algorithm for the saturated diagnostic classification model Psychometrika 85(4) 973–995 10.1007/s11336-020-09739-wCrossRefGoogle Scholar
Zhang, S., Chen, Y., (2022). Computation for latent variable model estimation: A unified stochastic proximal framework Psychometrika 10.1007/s11336-022-09863-9 35962849 9636119CrossRefGoogle ScholarPubMed
Supplementary material: File

Oka and Okada supplementary material

Oka and Okada supplementary material 1
Download Oka and Okada supplementary material(File)
File 305.2 KB
Supplementary material: File

Oka and Okada supplementary material

Oka and Okada supplementary material 2
Download Oka and Okada supplementary material(File)
File 891.4 KB