Hostname: page-component-745bb68f8f-g4j75 Total loading time: 0 Render date: 2025-01-08T12:09:54.535Z Has data issue: false hasContentIssue false
  • English
  • Français

Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing

Published online by Cambridge University Press:  01 January 2025

Wim J. van der Linden  and
Fanmin Guo
Wim J. van der Linden*
Affiliation:
University of Twente
Fanmin Guo
Affiliation:
Graduate Management Admission Council
*
Requests for reprints should be sent to Wim J. van der Linden, Department of Research Methodology, Measurement, and Data Analysis, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands. E-mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

In order to identify aberrant response-time patterns on educational and psychological tests, it is important to be able to separate the speed at which the test taker operates from the time the items require. A lognormal model for response times with this feature was used to derive a Bayesian procedure for detecting aberrant response times. Besides, a combination of the response-time model with a regular response model in an hierarchical framework was used in an alternative procedure for the detection of aberrant response times, in which collateral information on the test takers’ speed is derived from their response vectors. The procedures are illustrated using a data set for the Graduate Management Admission Test® (GMAT®). In addition, a power study was conducted using simulated cheating behavior on an adaptive test.

Keywords

adaptive testingBayesian predictive checkscheatingcollateral informationhierarchical modelingresponse times
Type
Theory and Methods
Information
Psychometrika , Volume 73 , Issue 3 , September 2008 , pp. 365 - 384
Copyright
Copyright © 2008 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The authors have relied upon data supplied by the Graduate Management Admission Council® (GMAC®) to conduct the independent research that forms the basis for the findings and conclusions stated in this article. These findings and conclusions are the opinion of the authors only, and do not necessarily reflect the opinion of the GMAC®. The authors are indebted to Wim M.M. Tielen and Rinke H. Klein Entink for their computational support.

References

Albert, J.H. (1992). Bayesian estimation of normal-ogive item response curves using Gibbs sampling. Journal of Educational and Behavioral Statistics, 17, 261–269.Google Scholar
Bradlow, E.T., Weiss, R.E., & Cho, M. (1998). Bayesian detection of outliers in computerized adaptive tests. Journal of the American Statistical Association, 93, 910–919.CrossRefGoogle Scholar
Casella, G., & Berger, R.L. (2002). Statistical inference, (2nd ed.). Pacific Grove: Duxbury.Google Scholar
Chang, H.-H., & Stout, W. (1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika, 58, 37–52.CrossRefGoogle Scholar
Fisher, R.A. (1925). Statistical methods for research workers, Edinburgh: Oliver & Boyd.Google Scholar
Gelman, A., Carlin, J.B, Stern, H., & Rubin, D.B. (1995). Bayesian data analysis, London: Chapman & Hall.CrossRefGoogle Scholar
Glas, C.A.W., & Meijer, R.R. (2003). A Bayesian approach to person fit analysis in item response theory models. Applied Psychological Measurement, 27, 217–233.CrossRefGoogle Scholar
Johnson, V.E., & Albert, J.H. (1999). Ordinal data modeling, New York: Springer.CrossRefGoogle Scholar
Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores, Reading: Addison-Wesley.Google Scholar
Meijer, R.R., & Sijtsma, K. (1995). Detection of aberrant item response patterns: A review of recent developments. Applied Measurement in Education, 8, 261–272.CrossRefGoogle Scholar
Meijer, R.R., & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107–135.CrossRefGoogle Scholar
Miller, G.A. (1956). The magic number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81–97.CrossRefGoogle ScholarPubMed
Owen, R.J. (1969). A Bayesian approach to tailored testing (Research Report 69-92). Princeton, NJ, Educational Testing Service.Google Scholar
Owen, R.J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of the American Statistical Association, 70, 351–356.CrossRefGoogle Scholar
van der Linden, W.J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31, 181–204.CrossRefGoogle Scholar
van der Linden, W.J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308.CrossRefGoogle Scholar
van der Linden, W.J. (2008). Using response times for item selection in adaptive tests. Journal of Educational and Behavioral Statistics, 33. In press.CrossRefGoogle Scholar
van der Linden, W.J., & van Krimpen-Stoop, E.M.L.A. (2003). Using response times to detect aberrant response patterns in computerized adaptive testing. Psychometrika, 68, 251–265.CrossRefGoogle Scholar
van der Linden, W.J., Scrams, D.J., & Schnipke, D.L. (1999). Using response-time constraints to control for speededness in computerized adaptive testing. Applied Psychological Measurement, 23, 195–210.CrossRefGoogle Scholar
van Krimpen-Stoop, E.M.L.A., & Meijer, R.R. (2001). CUSUM-based person fit statistics for adaptive testing. Journal of Educational and Behavioral Statistics, 26, 199–218.CrossRefGoogle Scholar