A procedure for maximizing the coefficient of generalizability under the constraint of limited resources is presented. The procedure uses optimization techniques that offer an investigator or test constructor the possibility of employing practical constraints. The procedure is illustrated for the two-facet random-model crossed design.

We provide a unified, theoretical basis on which measures of data reliability may be derived or evaluated, for both quantitative and qualitative data. This approach evaluates reliability as the “proportional reduction in loss” (PRL) that is attained in a sample by an optimal estimator. The resulting measure is between 0 and 1, linearly related to expected loss, and provides a direct way of contrasting the measured reliability in the sample with the least reliable and most reliable data-generating cases. The PRL measure is a generalization of many of the commonly-used reliability measures.

We show how the quantitative measures from generalizability theory can be derived as PRL measures (including Cronbach's alpha and measures proposed by Winer). For categorical data, we develop a new measure for the general case in which each of N judges assigns a subject to one of K categories and show that it is equivalent to a measure proposed by Perreault and Leigh for the case where N is 2.

Solutions for the problem of maximizing the generalizability coefficient under a budget constraint are presented. It is shown that the Cauchy-Schwarz inequality can be applied to derive optimal continuous solutions for the number of conditions of each facet.

We propose a class of confirmatory factor analysis models that include multiple sets of secondary or specific factors and a general factor. The general factor accounts for the common variance among manifest variables, whereas multiple sets of secondary factors account for the remaining source-specific dependency among subsets of manifest variables. A special case of the model is further proposed which constrains the specific factor loadings to be proportional to the general factor loadings. This proportional model substantially reduces the number of model parameters while preserving the essential structure of the general model. Furthermore, the proportional model allows for the interpretation of latent variables as the expected values of the observed manifest variables, decomposition of the variances, and the inclusion of interactions, similar to generalizability theory. We provide two applications to illustrate the utility of the proposed class of models.

The problem of predicting universe scores for samples of examinees based on their responses to samples of items is treated. A general measurement procedure is described in which multiple test forms are developed from a table of specifications and each form is administered to a different sample of examinees. The measurement model categorizes items according to the cells of such a table, and the linear function derived for minimizing error variance in prediction uses responses to these categories. In addition, some distinctions are drawn between aspects of the approach taken here and the familiar regressed score estimates.

Using the assumption of randomly parallel tests and concepts from generalizability theory, three signal/noise ratios for domain-referenced tests are developed, discussed, and compared. The three ratios have the same noise but different signals depending upon the kind of decision to be made as a result of measurement. It is also shown that these ratios incorporate a definition of noise or error which is different from the classical definition of noise typically used to characterize norm-referenced tests.

A two-facet measurement model with broad application in the behavioral sciences is identified, and its coefficient of generalizability (CG) is examined. A normalizing transformation is proposed, and an asymptotic variance expression is derived. Three other multifaceted measurement models and CGs are identified, and variance expressions are presented. Next, an empirical investigation of the procedures follows, and it is shown that, in most cases, Type I error control in inferential applications is precise, and that the estimates are relatively efficient compared with the correlation coefficient. Implications for further research and for practice are noted. In an Appendix, four additional models, CGs, and variance expressions are presented.

Many studies in fields such as psychology and educational sciences obtain information about attributes of subjects through observational studies, in which raters score subjects using multiple-item rating scales. Error variance due to measurement effects, such as items and raters, attenuate the regression coefficients and lower the power of (hierarchical) linear models. A modeling procedure is discussed to reduce the attenuation. The procedure consists of (1) an item response theory (IRT) model to map the discrete item responses to a continuous latent scale and (2) a generalizability theory (GT) model to separate the variance in the latent measurement into variance components of interest and nuisance variance components. It will be shown how measurements obtained from this mixture of IRT and GT models can be embedded in (hierarchical) linear models, both as predictor or criterion variables, such that error variance due to nuisance effects are partialled out. Using examples from the field of educational measurement, it is shown how general-purpose software can be used to implement the modeling procedure.

For some years, the limits of classic reliability theory have been recognized in favor of the Generalizability Theory, which deals simultaneously with multiple sources of error. This measurement model can be particularly useful when applied to research in cognitive psychophysiology. Indeed, studies in this field often deal with estimated measures whose reliability is rarely taken into account. In this paper, we report two generalizability studies in order to investigate the usefulness of G theory in providing information about the reliability of experimental results. The first was carried out on P300 measured during an oddball task, and the second was carried out on ERPs recorded during a recognition memory task. As expected, results showed that P300 modulation was more reliable than ERP memory modulation. This suggests that G theory can be a useful tool to estimate the reliability of psychophysiological findings, complementing and extending results from conventional analyses.

Temporal stability is an important fundamental quality when measuring sleep parameters, yet it has been infrequently assessed. Generalizability theory was used to estimate the short-term temporal stability of five variables commonly used to characterize insomnia: sleep onset latency, total sleep time, wake after sleep onset, time in bed, and sleep efficiency. Estimates were calculated for 32 elderly primary insomniacs and 32 elderly normal sleepers, both in the lab and at home, using both sleep logs and polysomnography (PSG). A week of recording using either PSG or sleep logs was typically sufficient to achieve adequate stability (defined as G coefficient of at least 0.80) with some notable exceptions: (a) when using log-derived measures with insomniacs, a 3-week average was necessary for wake after sleep onset and (b) more than a 2-week average was necessary for sleep onset latency. Because of the substantial commitment involved in the physiological recording of sleep, alternative forms of aggregation are considered with the intent of improving temporal stability.