How do neuropsychologists define cognitive constructs? Further thoughts on limitations of factor analysis used with normal or mixed clinical populations

In a recent study, we empirically demonstrated limitations in traditional ways that psychologists have used factor analysis to define cognitive constructs (Delis et al., 2003). Our criticism of factor analysis was not directed at this statistical method per se, but rather at how it has often been employed by psychologists to test cognitive constructs. Specifically, we pointed out shortcomings in using this technique with normal or mixed clinical populations. We argued that the factor-analytic studies of memory tests with normal or mixed clinical populations often yielded solutions in which measures of immediate and delayed memory loaded on the same factor. This particular use of factor analysis can mask important distinctions between critical cognitive functions that have been demonstrated using other research methods, such as experimental manipulations or case studies. We then conducted a factor-analytic study that empirically demonstrated that, whereas immediate and delayed memory measures load on the same factor when using normal or mixed clinical samples, these measures load on separate factors when using a homogenous population of patients with Alzheimer's disease (AD). We drew the conclusion that factor analytic techniques can still be used as one method for exploring conceptual relationships, but only if these methods are used as part of a systematic, programmatic exploration involving separate confirmatory factor analyses using multiple homogenous patient populations. In the first published reply to our study, Larrabee (2003) pointed out other limitations, stating that application of factor-analytic techniques to a single test that yields multiple measures may result in global, simplistic solutions due to method variance, which is the tendency of different variables from the same test to correlate significantly.

Bowden (2004) has written another reply addressing questions raised in the Delis et al. (2003) study. We applaud Bowden for further examining how neuropsychologists define cognitive constructs, because, as noted above, there is a critical need for this line of research in our field. We agree with several statements made by Bowden. He argues for more detailed factor structures, and an examination of aspects of the models not available from Principal Components Analysis. Bowden also emphasizes measurement invariance, and the use of vectors from observed score and latent variable means. Of considerable importance are his observations regarding apparent changes in between-group correlations due to sampling error, changes in score variability, or different trait compositions of observed scores in different groups. Bowden has thus provided an important list of complex issues that psychologists should address when using factor analysis to test the validity of cognitive constructs in our assessment instruments.

However, Bowden raised other points about which we have some questions. First, Bowden noted that, in a past study, he and his colleagues conducted a factor analytic study using “a large, diagnostically homogenous group of alcohol dependent patients,” and they nevertheless found support “that a distinction between immediate and delayed memory was not necessary” (see Bowden et al., 2001). One question we have about their study, however, concerns whether their sample of alcohol dependent patients was actually homogeneous. Specifically, this sample of 289 participants with alcohol abuse or dependency appeared to contain several major subgroups of patients. For example, the authors noted that about 40% of their sample had neurologic signs of Wernicke–Korsakoff syndrome (it is unclear whether the degree to which having “neurologic signs” constituted a formal diagnosis of alcoholic Korsakoff syndrome). The authors also indicated that “many of the participants suffered multiple organ disease,” but these diseases were not specified. In addition, the time when the cognitive tests were administered to these patients after they stopped drinking varied considerably in subgroups of the subjects, from 0–2 weeks at one site, to 4–6 weeks at a second site, and “at least 2 weeks after” at a third site. Duration and amount of alcohol intake varied widely in the sample as did the age range of 18–70 years. Taken together, Bowden et al.'s (2001) sample of subjects appears to represent a rather mixed clinical sample, the type of sample that, according to Delis et al. (2003), would be at risk for yielding factor solutions that mask critical distinctions between cognitive functions.

However, our central question for Bowden (2004) is that, how can one use only factor-analytic findings to draw the conclusion that “a distinction between immediate and delayed memory was not necessary”? Numerous experimental investigations of neurologic populations and classic case studies of amnesic patients have demonstrated striking dissociations between their scores on immediate and delayed memory measures (for a discussion, see Squire, 1987). These studies have demonstrated that it is critical for clinical memory tests to assess both immediate and delayed recall in order to characterize the full range of memory disorders across neurological populations.

A final question we have regarding Bowden's (2004) commentary concerns his criticism of our use of a homogenous sample of AD patients to conduct our factor-analytic study. Bowden felt that our use of AD patients constituted “criterion contamination,” because we selected these patients “on the basis of poor memory performance,” or based on specific “delayed memory scores.” According to Bowden, our use of specific patient groups “may not be representative of population scores on the memory variables.” First, the diagnoses of AD and HD were made independently by staff neurologists on the basis of history, the neurological and medical examinations, and mental status testing (i.e., the MMSE total score)—not on the basis of specific neuropsychological test results, including those from memory instruments. We are not exactly clear why selection based on known diagnostic status would be viewed as a problem. For decades, neuropsychologists have selected homogenous patient populations with relatively known cognitive profiles in order to test a priori hypotheses about cognitive dissociations.

There is a clear need for neuropsychologists to engage to a much greater degree in the types of analyses and discussions offered by Bowden (2004), with the ultimate goal of refining the methods we use to define our cognitive constructs. In our field, we have a number of methods available to formulate and test the validity of our cognitive constructs, including factor-analytic, experimental manipulation, and case-study approaches. As a field, we would enhance the rigor of our scientific and clinical methods by striving to derive a nomenclature of cognitive constructs based on multiple, empirically based methods that reveal a convergence of support for formulating distinctions in cognitive functions.