Item-level response time (RT) data can be conveniently collected from computer-based test/survey delivery platforms and have been demonstrated to bear a close relation to a miscellany of cognitive processes and test-taking behaviors. Individual differences in general processing speed can be inferred from item-level RT data using factor analysis. Conventional linear normal factor models make strong parametric assumptions, which sacrifices modeling flexibility for interpretability, and thus are not ideal for describing complex associations between observed RT and the latent speed. In this paper, we propose a semiparametric factor model with minimal parametric assumptions. Specifically, we adopt a functional analysis of variance representation for the log conditional densities of the manifest variables, in which the main effect and interaction functions are approximated by cubic splines. Penalized maximum likelihood estimation of the spline coefficients can be performed by an Expectation-Maximization algorithm, and the penalty weight can be empirically determined by cross-validation. In a simulation study, we compare the semiparametric model with incorrectly and correctly specified parametric factor models with regard to the recovery of data generating mechanism. A real data example is also presented to demonstrate the advantages of the proposed method.

Personality tests employing comparative judgments have been proposed as an alternative to Likert-type rating scales. One of the main advantages of a comparative format is that it can reduce faking of responses in high-stakes situations. However, previous research has shown that it is highly difficult to obtain trait score estimates that are both faking resistant and sufficiently accurate for individual-level diagnostic decisions. With the goal of contributing to a solution, I study the information obtainable from comparative judgments analyzed by means of Thurstonian IRT models. First, I extend the mathematical theory of ordinal comparative judgments and corresponding models. Second, I provide optimal test designs for Thurstonian IRT models that maximize the accuracy of people’s trait score estimates from both frequentist and Bayesian statistical perspectives. Third, I derive analytic upper bounds for the accuracy of these trait estimates achievable through ordinal Thurstonian IRT models. Fourth, I perform numerical experiments that complement results obtained in earlier simulation studies. The combined analytical and numerical results suggest that it is indeed possible to design personality tests using comparative judgments that yield trait scores estimates sufficiently accurate for individual-level diagnostic decisions, while reducing faking in high-stakes situations. Recommendations for the practical application of comparative judgments for the measurement of personality, specifically in high-stakes situations, are given.

This paper presents a machine learning approach to multidimensional item response theory (MIRT), a class of latent factor models that can be used to model and predict student performance from observed assessment data. Inspired by collaborative filtering, we define a general class of models that includes many MIRT models. We discuss the use of penalized joint maximum likelihood to estimate individual models and cross-validation to select the best performing model. This model evaluation process can be optimized using batching techniques, such that even sparse large-scale data can be analyzed efficiently. We illustrate our approach with simulated and real data, including an example from a massive open online course. The high-dimensional model fit to this large and sparse dataset does not lend itself well to traditional methods of factor interpretation. By analogy to recommender-system applications, we propose an alternative “validation” of the factor model, using auxiliary information about the popularity of items consulted during an open-book examination in the course.

Correspondence analysis (CA) is a statistical method for depicting the relationship between two categorical variables, and usually places an emphasis on graphical representations. In this study, we discuss a CA formulation based on canonical correlation analysis (CCA). In CCA-based formulation, the correlations within and between row/column categories in a reduced dimensional space can be expressed by canonical variables. However, in existing CCA-based formulations, only orthogonal rotation is permitted. Herein, we propose an alternative CCA-based formulation that permits oblique rotation. In the proposed formulation, the CA loss function can be defined as maximizing the generalized coefficient of determination, which is a measure of proximity between two variables. Simulation studies and real data examples are presented in order to demonstrate the benefits of the proposed formulation.

A number of empirically based Q-matrix validation methods are available in the literature, all of which were developed for cognitive diagnosis models (CDMs) involving dichotomous attributes. However, in many applications, it is more instructionally relevant to classify students into more than two categories (e.g., no mastery, basic mastery, and advanced mastery). To extend the practical utility of CDMs, methods for validating the Q-matrix for CDMs that measure polytomous attributes are needed. This study focuses on validating the Q-matrix of the generalized deterministic input, noisy, “and” gate model for polytomous attributes (pG-DINA). The pGDI, an extension of the G-DINA model discrimination index, is proposed for polytomous attributes. The pGDI serves as the basis of a validation method that can be used not only to identify potential misspecified q-entries, but also to suggest more appropriate attribute-level specifications. The theoretical properties of the pGDI are underpinned by several mathematical proofs, whereas its practical viability is examined using simulation studies covering various conditions. The results show that the method can accurately identify misspecified q-entries and suggest the correct attribute-level specifications, particularly when high-quality items are involved. The pGDI is applied to a proportional reasoning test that measures several polytomous attributes.

Hierarchical models are often considered to measure latent concepts defining nested sets of manifest variables. Therefore, by supposing a hierarchical relationship among manifest variables, the general latent concept can be represented by a tree structure where each internal node represents a specific order of abstraction for the latent concept measured. In this paper, we propose a new latent factor model called second-order disjoint factor analysis in order to model an unknown hierarchical structure of the manifest variables with two orders. This is a second-order factor analysis, which—respect to the second-order confirmatory factor analysis—is exploratory, nested and estimated simultaneously by maximum likelihood method. Each subset of manifest variables is modeled to be internally consistent and reliable, that is, manifest variables related to a factor measure “consistently” a unique theoretical construct. This feature implies that manifest variables are positively correlated with the related factor and, therefore, the associated factor loadings are constrained to be nonnegative. A cyclic block coordinate descent algorithm is proposed to maximize the likelihood. We present a simulation study that investigates the ability to get reliable factors. Furthermore, the new model is applied to identify the underlying factors of well-being showing the characteristics of the new methodology. A final discussion completes the paper.

Latent variable models have been playing a central role in psychometrics and related fields. In many modern applications, the inference based on latent variable models involves one or several of the following features: (1) the presence of many latent variables, (2) the observed and latent variables being continuous, discrete, or a combination of both, (3) constraints on parameters, and (4) penalties on parameters to impose model parsimony. The estimation often involves maximizing an objective function based on a marginal likelihood/pseudo-likelihood, possibly with constraints and/or penalties on parameters. Solving this optimization problem is highly non-trivial, due to the complexities brought by the features mentioned above. Although several efficient algorithms have been proposed, there lacks a unified computational framework that takes all these features into account. In this paper, we fill the gap. Specifically, we provide a unified formulation for the optimization problem and then propose a quasi-Newton stochastic proximal algorithm. Theoretical properties of the proposed algorithms are established. The computational efficiency and robustness are shown by simulation studies under various settings for latent variable model estimation.

Equal parameter estimates across subgroups is a substantial requirement of statistical tests. Ignoring subgroup differences poses a threat to study replicability, model specification, and theory development. Structural change tests are a powerful statistical technique to assess parameter invariance. A core element of those tests is the empirical fluctuation process. In the case of parameter invariance, the fluctuation process asymptotically follows a Brownian bridge. This asymptotic assumption further provides the basis for inference. However, the empirical fluctuation process does not follow a Brownian bridge in small samples, and this situation is amplified in large psychometric models. Therefore, common methods of obtaining the sampling distribution are invalid and the structural change test becomes conservative. We discuss an alternative solution to obtaining the sampling distribution—permutation approaches. Permutation approaches estimate the sampling distribution through resampling of the dataset, avoiding distributional assumptions. Hereby, the tests power are improved. We conclude that the permutation alternative is superior to standard asymptotic approximations of the sampling distribution.

Recently, machine learning (ML) methods have been used in causal inference to estimate treatment effects in order to reduce concerns for model mis-specification. However, many ML methods require that all confounders are measured to consistently estimate treatment effects. In this paper, we propose a family of ML methods that estimate treatment effects in the presence of cluster-level unmeasured confounders, a type of unmeasured confounders that are shared within each cluster and are common in multilevel observational studies. We show through simulation studies that our proposed methods are robust from biases from unmeasured cluster-level confounders in a variety of multilevel observational studies. We also examine the effect of taking an algebra course on math achievement scores from the Early Childhood Longitudinal Study, a multilevel observational educational study, using our methods. The proposed methods are available in the CURobustML R package.

The paper outlines several approaches for dealing with meta-analyses of count outcome data. These counts are the accumulation of occurred events, and these events might be rare, so a special feature of the meta-analysis is dealing with low counts including zero-count studies. Emphasis is put on approaches which are state of the art for count data modelling including mixed log-linear (Poisson) and mixed logistic (binomial) regression as well as nonparametric mixture models for count data of Poisson and binomial type. A simulation study investigates the performance and capability of discrete mixture models in estimating effect heterogeneity. The approaches are exemplified on a meta-analytic case study investigating the acceptance of bibliotherapy.

Component-based approaches have been regarded as a tool for dimension reduction to predict outcomes from observed variables in regression applications. Extended redundancy analysis (ERA) is one such component-based approach which reduces predictors to components explaining maximum variance in the outcome variables. In many instances, ERA can be extended to capture nonlinearity and interactions between observed and components, but only by specifying a priori functional form. Meanwhile, machine learning methods like neural networks are typically used in a data-driven manner to capture nonlinearity without specifying the exact functional form. In this paper, we introduce a new method that integrates neural networks algorithms into the framework of ERA, called NN-ERA, to capture any non-specified nonlinear relationships among multiple sets of observed variables for constructing components. Simulations and empirical datasets are used to demonstrate the usefulness of NN-ERA. The conclusion is that in social science datasets with unstructured data, where we expect nonlinear relationships that cannot be specified a priori, NN-ERA with its neural network algorithmic structure can serve as a useful tool to specify and test models otherwise not captured by the conventional component-based models.

In this paper, we propose a model-based method to study conditional dependence between response accuracy and response time (RT) with the diffusion IRT model (Tuerlinckx and De Boeck in Psychometrika 70(4):629–650, 2005, https://doi.org/10.1007/s11336-000-0810-3; van der Maas et al. in Psychol Rev 118(2):339–356, 2011, https://doi.org/10.1080/20445911.2011.454498). We extend the earlier diffusion IRT model by introducing variability across persons and items in cognitive capacity (drift rate in the evidence accumulation process) and variability in the starting point of the decision processes. We show that the extended model can explain the behavioral patterns of conditional dependency found in the previous studies in psychometrics. Variability in cognitive capacity can predict positive and negative conditional dependency and their interaction with the item difficulty. Variability in starting point can account for the early changes in the response accuracy as a function of RT given the person and item effects. By the combination of the two variability components, the extended model can produce the curvilinear conditional accuracy functions that have been observed in psychometric data. We also provide a simulation study to validate the parameter recovery of the proposed model and present two empirical applications to show how to implement the model to study conditional dependency underlying data response accuracy and RTs.

Risk behavior has substantial consequences for health, well-being, and general behavior. The association between real-world risk behavior and risk behavior on experimental tasks is well documented, but their modeling is challenging for several reasons. First, many experimental risk tasks may end prematurely leading to censored observations. Second, certain outcome values can be more attractive than others. Third, a priori unknown groups of participants can react differently to certain risk-levels. Here, we propose the censored mixture model which models risk taking while dealing with censoring, attractiveness to certain outcomes, and unobserved individual risk preferences, next to experimental conditions.

Major depression is a severe mental disorder that is associated with strongly increased mortality. The quantification of its prevalence on regional levels represents an important indicator for public health reporting. In addition to that, it marks a crucial basis for further explorative studies regarding environmental determinants of the condition. However, assessing the distribution of major depression in the population is challenging. The topic is highly sensitive, and national statistical institutions rarely have administrative records on this matter. Published prevalence figures as well as available auxiliary data are typically derived from survey estimates. These are often subject to high uncertainty due to large sampling variances and do not allow for sound regional analysis. We propose a new area-level Poisson mixed model that accounts for measurement errors in auxiliary data to close this gap. We derive the empirical best predictor under the model and present a parametric bootstrap estimator for the mean squared error. A method of moments algorithm for consistent model parameter estimation is developed. Simulation experiments are conducted to show the effectiveness of the approach. The methodology is applied to estimate the major depression prevalence in Germany on regional levels crossed by sex and age groups.

Algorithmic automatic item generation can be used to obtain large quantities of cognitive items in the domains of knowledge and aptitude testing. However, conventional item models used by template-based automatic item generation techniques are not ideal for the creation of items for non-cognitive constructs. Progress in this area has been made recently by employing long short-term memory recurrent neural networks to produce word sequences that syntactically resemble items typically found in personality questionnaires. To date, such items have been produced unconditionally, without the possibility of selectively targeting personality domains. In this article, we offer a brief synopsis on past developments in natural language processing and explain why the automatic generation of construct-specific items has become attainable only due to recent technological progress. We propose that pre-trained causal transformer models can be fine-tuned to achieve this task using implicit parameterization in conjunction with conditional generation. We demonstrate this method in a tutorial-like fashion and finally compare aspects of validity in human- and machine-authored items using empirical data. Our study finds that approximately two-thirds of the automatically generated items show good psychometric properties (factor loadings above .40) and that one-third even have properties equivalent to established and highly curated human-authored items. Our work thus demonstrates the practical use of deep neural networks for non-cognitive automatic item generation.

Process data refer to data recorded in computer-based assessments (CBAs) that reflect respondents’ problem-solving processes and provide greater insight into how respondents solve problems, in addition to how well they solve them. Using the rich information contained in process data, this study proposed an item expansion method to analyze action-level process data from the perspective of diagnostic classification in order to comprehensively understand respondents’ problem-solving competence. The proposed method cannot only estimate respondents’ problem-solving ability along a continuum, but also classify respondents according to their problem-solving skills. To illustrate the application and advantages of the proposed method, a Programme for International Student Assessment (PISA) problem-solving item was used. The results indicated that (a) the estimated latent classes provided more detailed diagnoses of respondents’ problem-solving skills than the observed score categories; (b) although only one item was used, the estimated higher-order latent ability reflected the respondents’ problem-solving ability more accurately than the unidimensional latent ability estimated from the outcome data; and (c) interactions among problem-solving skills followed the conjunctive condensation rule, which indicated that the specific action sequence appeared only when a respondent mastered all required problem solving skills. In conclusion, the proposed diagnostic classification approach is feasible and promising analyzing process data.

In practice, it is common that a best fitting structural equation model (SEM) is selected from a set of candidate SEMs and inference is conducted conditional on the selected model. Such post-selection inference ignores the model selection uncertainty and yields too optimistic inference. Using the largest candidate model avoids model selection uncertainty but introduces a large variation. Jin and Ankargren (Psychometrika 84:84–104, 2019) proposed to use frequentist model averaging in SEM with continuous data as a compromise between model selection and the full model. They assumed that the true values of the parameters depend on $n^{-1/2}$\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n^{-1/2}$$\end{document} with n being the sample size, which is known as a local asymptotic framework. This paper shows that their results are not directly applicable to SEM with ordinal data. To address this issue, we prove consistency and asymptotic normality of the polychoric correlation estimators under the local asymptotic framework. Then, we propose a new frequentist model averaging estimator and a valid confidence interval that are suitable for ordinal data. Goodness-of-fit test statistics for the model averaging estimator are also derived.

The multidimensional forced-choice (MFC) format has been proposed to reduce faking because items within blocks can be matched on desirability. However, the desirability of individual items might not transfer to the item blocks. The aim of this paper is to propose a mixture item response theory model for faking in the MFC format that allows to estimate the fakability of MFC blocks, termed the Faking Mixture model. Given current computing capabilities, within-subject data from both high- and low-stakes contexts are needed to estimate the model. A simulation showed good parameter recovery under various conditions. An empirical validation showed that matching was necessary but not sufficient to create an MFC questionnaire that can reduce faking. The Faking Mixture model can be used to reduce fakability during test construction.

A mixed-effects location scale model allows researchers to study within- and between-person variation in repeated measures. Key components of the model include separate variance models to study predictors of the within-person variance, as well as predictors of the between-person variance of a random effect, such as a random intercept. In this paper, a latent variable mixed-effects location scale model is developed that combines a longitudinal common factor model and a mixed-effects location scale model to characterize within- and between-person variation in a common factor. The model is illustrated using daily reports of positive affect and daily stressors for a large sample of adult women.