Hostname: page-component-745bb68f8f-grxwn Total loading time: 0 Render date: 2025-01-07T18:06:25.608Z Has data issue: false hasContentIssue false
  • English
  • Français

Modeling Differences Between Response Times of Correct and Incorrect Responses

Published online by Cambridge University Press:  01 January 2025

Maria Bolsinova  and
Jesper Tijmstra
Maria Bolsinova*
Affiliation:
ACTNext
Jesper Tijmstra
Affiliation:
Tilburg University
*
Correspondence should be made to Maria Bolsinova, ACTNext, 500 ACT dr., Iowa City, IA 52243, USA. Email:[email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

While standard joint models for response time and accuracy commonly assume the relationship between response time and accuracy to be fully explained by the latent variables of the model, this assumption of conditional independence is often violated in practice. If such violations are present, taking these residual dependencies between response time and accuracy into account may both improve the fit of the model to the data and improve our understanding of the response processes that led to the observed responses. In this paper, we propose a framework for the joint modeling of response time and accuracy data that allows for differences in the processes leading to correct and incorrect responses. Extensions of the standard hierarchical model (van der Linden in Psychometrika 72:287–308, 2007. https://doi.org/10.1007/s11336-006-1478-z) are considered that allow some or all item parameters in the measurement model of speed to differ depending on whether a correct or an incorrect response was obtained. The framework also allows one to consider models that include two speed latent variables, which explain the patterns observed in the responses times of correct and of incorrect responses, respectively. Model selection procedures are proposed and evaluated based on a simulation study, and a simulation study investigating parameter recovery is presented. An application of the modeling framework to empirical data from international large-scale assessment is considered to illustrate the relevance of modeling possible differences between the processes leading to correct and incorrect responses.

Keywords

conditional dependencehierarchical modeljoint modelingresponse times
Type
Original Paper
Information
Psychometrika , Volume 84 , Issue 4 , December 2019 , pp. 1018 - 1046
Copyright
Copyright © 2019 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Electronic supplementary materialThe online version of this article (https://doi.org/10.1007/s11336-019-09682-5) contains supplementary material, which is available to authorized users.

References

Albert, J.. (1992). Bayesian estimation of Normal Ogive item response curves using Gibbs sampling. Journal of Educational Statistics, 17 251– 269. CrossRefGoogle Scholar
Birnbaum, A., Lord, F. M., &Novick, M. R.. (1968). Some latent trait models and their use in inferring an examinee’s ability. Statistical theories of mental test scores, Reading: Addison-Wesley. 395– 479. Google Scholar
Bolsinova, M.,De Boeck, P., &Tijmstra, J.. (2017). Modelling conditional dependence between response time and accuracy. Psychometrika, 82 (4) 1126– 1148. CrossRefGoogle ScholarPubMed
Bolsinova, M., &Tijmstra, J.. (2016). Posterior predictive checks for conditional independence between response time and accuracy. Journal of Educational and Behavioral Statistics, 41 (2) 123– 145. CrossRefGoogle Scholar
Bolsinova, M., &Tijmstra, J.. (2018). Improving precision of ability estimation: Getting more from response times. British Journal of Mathematical and Statistical Psychology, 71 (1) 13– 38. CrossRefGoogle ScholarPubMed
Bolsinova, M.,Tijmstra, J., &Molenaar, D.. (2017). Response moderation models for conditional dependence between response time and response accuracy. British Journal of Mathematical and Statistical Psychology, 70 (2) 257– 279. CrossRefGoogle ScholarPubMed
Bolsinova, M., Tijmstra, J., Molenaar, D., & De Boeck, P. (2017b). Conditional dependence between response time and accuracy: An overview of its possible sources and directions for distinguishing between them. Frontiers in Psychology, 8, 202. https://doi.org/10.3389/fpsyg.2017.00202.CrossRefGoogle Scholar
Gelman, A.,Meng, X., &Stern, H.. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 4 733– 807. Google Scholar
Goldhammer, F., Martens, T., Christoph, G., & Lüdtke, O. (2016). Test-taking engagement in PIAAC. OECD Education Working Papers, (133). https://doi.org/10.1787/5jlzfl6fhxs2-en CrossRefGoogle Scholar
Hoff, P. D.(2009). A first course in Bayesian statistical methods, New York: Spinger. CrossRefGoogle Scholar
Klein Entink, R.,Fox, J., &van der Linden, W. J.. (2009). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74 21– 48. CrossRefGoogle Scholar
Klein Entink, R.,van der Linden, W. J., &Fox, J-. P.. (2009). A Box–Cox normal model for response times. British Journal of Mathematical and Statistical Psychology, 62 (3) 621– 640. CrossRefGoogle ScholarPubMed
Lord, F., &Novick, M.. (1968). Statistical theories of mental test scores, Reading, MA: Addison-Wesley. Google Scholar
Meng, X. B.,Tao, J., &Chang, H. H.. (2015). A conditional joint modeling approach for locally dependent item responses and response times. Journal of Educational Measurement, 52 1– 27. CrossRefGoogle Scholar
Meng, X. -L.. (1994). Posterior predictive p-values. The Annals of Statistics, 22 (3) 1142– 1160. CrossRefGoogle Scholar
Molenaar, D.,Tuerlinckx, F., &van der Maas, H.. (2015). A bivariate generalized linear item response theory modeling framework to the analysis of responses and response times. Multivariate Behavioural Research, 50 56– 74. CrossRefGoogle Scholar
Molenaar, D.,Tuerlinckx, F., &van der Maas, H.. (2015). A generalized linear factor model approach to the hierarchical framework for responses and response times. British Journal of Mathematical and Statistical Psychology, 68 197– 219. CrossRefGoogle Scholar
OECD. (2012). Literacy, numeracy and problem solving in technology-rich environments. Paris: OECD Publishing. https://doi.org/10.1787/9789264128859-en.CrossRefGoogle Scholar
Partchev, I., &De Boeck, P.. (2012). Can fast and slow intelligence be differentiated?. Intelligence, 40 23– 32. CrossRefGoogle Scholar
Ranger, J., &Ortner, T.. (2012). The case of dependency of responses and response times: A modeling approach based on standard latent trait models. Psychological Test and Assessment Modeling, 54 128– 148. Google Scholar
Rouder, J. N.,Sun, D.,Speckman, P. L.,Lu, J., &Zhou, D.. (2003). A hierarchical bayesian statistical framework for response time distributions. Psychometrika, 68 (4) 589– 606. CrossRefGoogle Scholar
Scherer, R.,Greiff, S., &Hautamäki, J.. (2015). Exploring the relation between time on task and ability in complex problem solving. Intelligence, 48 37– 50. CrossRefGoogle Scholar
Semmes, R.,Davison, M. L., &Close, C.. (2011). Modeling individual differences in numerical reasoning speed as a random effect of response time limits. Applied Psychological Measurement, 35 (6) 433– 446. CrossRefGoogle Scholar
Tanner, M., &Wong, W.. (1987). The calculation of posterior distributions by data augmentation. Journal of the American Statistical Association, 82 528– 540. CrossRefGoogle Scholar
van der Linden, W. J.. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioural Statistics, 31 181– 204. Google Scholar
van der Linden, W. J.. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72 287– 308. CrossRefGoogle Scholar
van der Linden, W. J.. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46 (3) 247– 272. CrossRefGoogle Scholar
van der Linden, W. J., &Glas, CAW.. (2010). Statistical tests of conditional independence between responses and/or response times on test items. Psychometrika, 75 (1) 120– 139. CrossRefGoogle Scholar
van der Linden, W. J.,Klein Entink, R. H., &Fox, J. P.. (2010). IRT parameter estimation with response times as collateral information. Applied Psychological Measurement, 34 327– 347. CrossRefGoogle Scholar
van der Maas, H. L., &Jansen, B. R.. (2003). What response times tell of children’s behavior on the balance scale task. Journal of Experimental Child Psychology, 85 (2) 141– 177. CrossRefGoogle ScholarPubMed
van der Maas, H. L., &Wagenmakers, E-. J.. (2005). A psychometric analysis of chess expertise. The American journal of psychology, 118 29– 60. CrossRefGoogle ScholarPubMed
Van de Schoot, R.,Lugtig, P., &Hox, J.. (2012). A checklist for testing measurement invariance. European Journal of Developmental Psychology, 9 (4) 486– 492. CrossRefGoogle Scholar
Wang, C.,Chang, HH., &Douglas, JA.. (2013). The linear transformation model with frailties for the analysis of item response times. British Journal of Mathematical and Statistical Psychology, 66 (1) 144– 168. CrossRefGoogle ScholarPubMed
Wang, C.,Fan, Z.,Chang, H. H., &Douglas, J. A.. (2013). A semiparametric model for jointly analyzing response times and accuracy in computerized testing. Journal of Educational and Behavioral Statistics, 38 (4) 381– 417. CrossRefGoogle Scholar
Zhan, P.,Jiao, H., &Liao, D.. (2017). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71 262– 286. CrossRefGoogle ScholarPubMed
Supplementary material: File

Bolsinova and Tijmstra supplementary material

Bolsinova and Tijmstra supplementary material
Download Bolsinova and Tijmstra supplementary material(File)
File 150.8 KB