Canadian Normative Data for Minimal Assessment of Cognitive Function in Multiple Sclerosis

Abstract

Objective: The Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS) is a consensus-based collection of neuropsychological tests that evaluate cognitive functioning in individuals with multiple sclerosis (MS). The tests are typically scored using each respective published test manual, leaving the examiner to make interpretations from norms derived from different American populations. Given demographic differences, this may lead to misinterpretation of findings in Canadians. Our goal was to establish both discrete and regression-based normative data for the MACFIMS based on a largely co-normed Canadian population to allow for improved psychometric interpretation. Methods: MACFIMS data sets were aggregated from across three different Canadian cities (Ottawa, Toronto, and London), yielding a total of 330 healthy control participants from four different studies evaluating cognition in individuals with MS. Given the variety of contributing studies, there was variability in terms of the number of participants completing each measure. Results: Both age-based discrete normative data and demographically adjusted (sex, age, and education) regression-based formulae were established. The demographic variables varied in their contribution to each MACFIMS test in the regression models, predicting 0 to 18% of the variance. Conclusions: Provision of these regression-based formulae will allow for more accurate interpretation of Canadian-derived MACFIMS scores by allowing clinicians to correct for all relevant demographic variables simultaneously, leading to improved clinical decision making for individuals with multiple sclerosis.

Résumé

Données normatives canadiennes pour leMinimal Assessment of Cognitive Function in Multiple Sclerosis. Objectif: Le Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS) est un ensemble de tests neuropsychologiques, basé sur un consensus, qui évaluent la fonction cognitive d’individus atteints de sclérose en plaques (SP). Les tests sont habituellement notés selon les directives contenues dans les manuels publiés pour chacun des tests, laissant l’examinateur en faire l’interprétation à partir de normes provenant de différentes populations américaines. Étant donné l’existence de différences démographiques entre différentes populations, cette façon de procéder peut mener à une interprétation erronée des observations chez des sujets canadiens. Le but de notre étude était d’établir des données normatives discrètes ainsi que des données s’appuyant sur une analyse de régression pour le MACFIMS, basées sur une population canadienne de référence, afin d’en améliorer l’interprétation psychométrique. Méthodologie: Des ensembles de données du MACFIMS provenant de trois villes canadiennes (Ottawa, Toronto et London) ont été regroupés, soit au total les données de 330 sujets témoins en bonne santé qui avaient participé à 4 études différentes évaluant la fonction cognitive d’individus atteints de SP. Étant donné la diversité des études, le nombre de participants qui avaient complété chaque mesure pouvait varier. Résultats: Nous avons établi des formules basées sur des données normatives discrètes, selon l’âge, et sur des données ajustées au point de vue démographique (le sexe, l’âge et le niveau de scolarité) à l’analyse de régression. La contribution des variables démographiques à chaque test du MACFIMS variait dans les modèles de régression, prédisant de 0 à 18% de la variance. Conclusions: Des formules basées sur l’analyse de régression permettront une interprétation plus juste des scores canadiens au MACFIMS parce que les cliniciens pourront corriger simultanément les données pour toutes les variables démographiques pertinentes, ce qui entraînera une amélioration des décisions cliniques concernant les patients atteints de MS.

A number of different assessment tools have been proposed over the past few decades to evaluate cognition in multiple sclerosis (MS). ^{1 - 4} However, the most commonly accepted comprehensive battery of tests for neuropsychological evaluation in MS is the Minimal Assessment of Cognitive Function in MS (MACFIMS). ⁵ This battery of tests was chosen via consensus by an expert panel of neuropsychologists and psychologists (with clinical and research expertise in MS) from the United States, Canada, United Kingdom, and Australia at a 2001 meeting sponsored by the Consortium of MS Centers (CMSC). ⁵ Following a review of the pertinent literature, criteria were established with which to select appropriate measures. These criteria included: standardized stimulus materials and administration, normative data, adequate range, reliability, criterion validity, alternate forms and practicality. ⁵ In addition, the measures chosen had to reflect the domains of cognition typically affected in MS: processing speed/working memory, language, visual perception/spatial processing, learning and memory, and executive function. The following measures were selected: the Paced Auditory Serial Addition Test (Rao version; PASAT); ^{6 , 7} the oral Symbol Digit Modalities Test (Rao version; SDMT); ^{6 , 8} the Controlled Oral Word Association Test; ⁹ the Judgment of Line Orientation Test (JLO); ¹⁰ the California Verbal Learning Test–II (CVLT–II); ¹¹ the Brief Visuospatial Memory Test–Revised (BVMT–R); ¹² and the Delis–Kaplan Executive Function System (D–KEFS) Sorting Test (Sorting). ¹³ This battery has been well-validated in MS and is used in common practice by clinicians assessing cognition in MS. ¹⁴ Components of the MACFIMS battery (i.e., the BVMT–R) are more sensitive to impairment than other established batteries ¹⁵ and are predictive of health-related quality of life (i.e., the SDMT). ¹⁶ MACFIMS data are strongly related to the neuropsychiatric features of the disease, neuroimaging, and vocational outcomes. ^{17 - 19}

When deciding whether or not a patient is cognitively impaired on the MACFIMS, clinicians typically consult normative data for each respective test. These data are provided in all published test manuals, allowing clinicians to compare patients’ performance relative to that of demographically similar healthy individuals. Typically, performance is considered impaired if a person scores at, or more than, 1.5 standard deviations below the normative mean. Given that age affects performance on most cognitive tasks, ²⁰ normative data are typically supplied in predetermined, and sometimes arbitrary, age groupings. Performance may also be affected by educational level and gender, and thus some normative data take these variables into account as well. ^{21 , 22} Such discrete norms are typically used in common practice by clinicians. ^{23 - 25}

More recently, there is a movement toward regression-based or continuous norms. ²⁶ Data are derived from regression-based formulae in which numerous demographic variables can be included (e.g., age, education, sex, race, IQ, and physical health). ^{27 , 28} Regression-based norms are not vulnerable to the same limitations plaguing discrete norms such as arbitrary cutoffs for the age/education groupings (which can affect interpretation depending on which age category a person is assigned to), ²⁹ small numbers of participants in individual subgroups, and the lack of correction for all relevant demographic information. ³⁰ In addition, typically smaller sample sizes are needed in regression-based normative data to obtain norms as precise as those provided by discrete methods. ²⁹

To the best of our knowledge, although several studies have presented control data on the MACFIMS when comparing a full control sample to an MS sample, ^{14 , 31 , 32} these data have not been presented in a manner that would allow their use for normative comparison in a clinical setting. Indeed, the control data are presented as a group, rather than broken down into age or other demographically relevant categories. Thus, there are no clinically relevant discrete norms available for the MACFIMS on a largely co-normed population. In other words, although discrete published normative data are available for all components of the MACFIMS, each particular test was normed on different populations. Although some researchers suggest that co-norming is not required for competent clinical practice, ³³ others stress that norms derived from different populations cannot be considered equivalent given differences in demographics between the various normative populations from which individual published tests were derived. ³⁴ Without co-norming, there is room for error in interpretation given that the populations are not likely to be equivalent. Despite this, clinicians are increasingly relying on flexible batteries, so that in practice co-norming is not considered a priority. ³³ Nonetheless, given the psychometric advantage of co-norming, having such data available for the MACFIMS would be an asset.

In the area of cognitive assessment in MS, regression-based normative data are used with the National MS Society Consensus Neuropsychological Battery for Pediatric Multiple Sclerosis (NBPMS) ^{35 , 36} and have also been derived for the MACFIMS in an American sample. ³⁷ The latter was demonstrated to yield higher rates of impairment than manualised norms for many of the MACFIMS measures when evaluating people with MS. Although members of our group have published regression-based normative data on the SDMT in a Canadian sample ³⁰ (and the Ottawa control data from that study are also included in the current paper), to the best of our knowledge, no such Canadian normative data are available on the full MACFIMS battery. There is a need for such data given that it has been demonstrated that Canadian adults attain higher raw scores than Americans on measures of general intellectual ability as evaluated by the Wechsler Adult Intelligence Scale (versions III and IV), ³⁸ which has been attributed to variations in population composition due to social, economic, and educational differences between the two countries. ³⁹ When assessing a person with an acquired neurological condition, clinicians may conclude that the person is more significantly impaired when using Canadian norms relative to American ones. ³⁹ Others suggest that education level has a stronger relationship with IQ in American samples compared to Canadian ones. ⁴⁰ Miller et al. (2015) ³⁸ concluded that “choosing to use U.S. norms to interpret the test scores of a Canadian will increase the variance and subsequently increase the [confidence intervals] around the true score, making it more difficult to rely on the test scores for diagnosis” (P. 323–4). This highlights the need for Canadian MACFIMS normative data and is reflective of best practice recommendations. ³⁸ MS researchers from other countries are also recognizing the need for culturally sensitive use of MS-focused testing batteries such as the MACFIMS ^{31 , 32} and the Brief International Cognitive Assessment for MS (BICAMS). ^{41 - 44}

Our current objective was to establish both discrete and regression-based normative data for the MACFIMS based on a partially co-normed Canadian population.

Methods

The studies that contributed to this database were approved by appropriate institutional review boards, which included the Ottawa Health Science Network Research Ethics Board, the Sunnybrook Health Sciences Centre Research Ethics Board, and the Health Sciences Research Ethics Board (London).

Participants

Research participants were healthy controls from three medium to large Ontario cities (Ottawa, London, and Toronto). Participants were recruited from the community, through word of mouth, via posted advertisements, and using website advertisements. All participating subjects provided full informed consent. The data were collected in the course of four different studies (each with different original aims) evaluating various aspects of cognitive functioning in persons with multiple sclerosis (see Supplementary Table 1). Overall, 174 participants were obtained from studies conducted by Dr. Walker in Ottawa. The Toronto site enrolled six healthy controls. Lastly, there were 150 healthy controls from London who completed the SDMT and PASAT. ⁴⁵ Together, there was a grand total of 330 healthy control participants, although the number completing each individual MACFIMS test varied as indicated.

The inclusion criteria were consistent across the contributing studies. In order to be included, participants had to be healthy, between 18 and 65 years of age (18 and 59 for the London sample), and fluent in English (according to both participant self-assessment and examiner assessment). Participants were excluded if they had any neurological/medical/psychiatric conditions that might impede cognition, including prior head trauma, learning disability, attention-deficit disorder, mild cognitive impairment, dementia, or substance abuse. Other exclusion criteria included a history of seizures, uncorrected visual acuity problems, and current use of drugs (legal or illegal) that might have an impact on cognitive function.

Neuropsychological Measures

The MACFIMS ^{5 , 14} was administered across the four different studies by trained assistants and students under the supervision of either a neuropsychologist (LW), a neuropsychiatrist (AF), or a neurologist with expertise in cognition in MS (SM). The full MACFIMS [PASAT (3” and 2”), oral SDMT, verbal fluency (FAS and Animal Naming substituted for COWAT), ⁴⁶ the JLO, CVLT–II, BVMT–R, and D–KEFS Sorting] was administered in the SUNSCREEN study. In the IPSIMS study, participants were administered all MACFIMS tests but the JLO and CVLT–II. In the BICAMS study, participants were administered all MACFIMS tests but D–KEFS Sorting and JLO. In addition, they received only the learning trials from the CVLT–II and the BVMT–R. The London participants completed only the SDMT and PASAT.

Data Analysis

Discrete normative data were established by first subdividing the sample according to three age groupings: 18-35, 36-50, and 51-65. Groupings were chosen to ensure the maximum number of participants in each cell, as well as to ensure that the data would be available for evaluating the performance of young, middle-aged, and older adults. Descriptive statistics (means and standard deviations) were calculated for each group on each measure.

For the regression-based norms, a series of linear regressions were calculated according to a procedure already well-established in the literature. ⁴⁷ In most instances, just the summary score for the measure was employed, but for the CVLT–II and the BVMT–R, additional analyses were performed for relevant sub-scores. In each regression model, raw scores were the criterion variables, and age (in years), gender (coded as 1=male, 2=female), and education (in years) were the predictor variables entered as a single block (i.e., the “Enter” method). Given that multiple regression models were examined, a p value <0.01 was utilized throughout. As in Duff & Ramezani (2015), ⁴⁷ for each regression model, the constant and non-standardized coefficients were used to generate normative formulae for ease of clinical use.

Results

See Table 1 for demographic information broken down by individual cognitive tests. In addition, for further reference, Supplementary Table 1 lists demographics for the full sample broken down by study/location. In the final sample of 330 participants, the mean age was 39.78 (11.78) years, most were female (74 vs. 26% male) given that they were selected to match the female-dominant MS populations in their respective studies, and the mean level of education was equivalent to a community college education (~15 years). Mean scores on the MACFIMS tasks fell generally within the average range when compared to published normative data for each test, suggesting that our sample was representative of the larger population.

Table 1 MACFIMS tests and associated sample sizes, age and education

Discrete Norms

See Table 2 for age-based discrete normative data.

Table 2 MACFIMS discrete norms for Canadian sample

^* Must reverse direction of sign on resulting z score.

Regression-Based Norms

See Table 3 for individual regression-based formulae associated with each MACFIMS test. Demographic variables varied in their contribution to each MACFIMS test in the regression models, predicting 0-18% of the variance, with a mean of 7% of the variance in the MACFIMS test scores being explained by these demographic variables. The only regression equations that failed to reach statistical significance were for the FAS and Animal Fluency (although they approached significance) and the CVLT–II, primarily because in each of these equations demographics accounted for less than 4% of the variance (on the primary measures).

Table 3 MACFIMS regression-based norms for Canadian sample

Sex: 1=male, 2=female; age: enter age in years; educ: enter education in years.

^* Must reverse direction of sign on resulting z score.

The following tests and subtests were more influenced by demographic variables as measured with standardized β weights (see Supplementary Table 2) greater than 0.20 (specific demographic variable listed in parentheses): SDMT (age); 2s PASAT (education); JLO (sex, age); CVLT–II Free Recall (education); CVLT–II List B Free Recall (education); CVLT–II Short Delay Free Recall (age, education); CVLT–II Long Delay Free Recall (age, education); CVLT–II Long Delay Cued Recall (age); CVLT–II, Total Intrusions (age, education); CVLT–II, Total Repetitions (age, education); BVMT–R Total Recall (age, education); BVMT–R Delay Recall (age); D–KEFS Confirmed Correct Sorts (sex); and D–KEFS Free Sort Description Score (sex).

Discussion

This project aimed to establish Canadian discrete and regression-based normative data for the MACFIMS battery within a partially co-normed framework. Although there were four studies contributing to the healthy control data derived here, and thus not completely co-normed, the majority of tests in the battery were completed by more than 100 participants. Although regression-based data are available for the MACFIMS derived from an American population, research has emphasized the inadvisability of using American normative data to interpret cognitive test findings from Canadian patients, at least in part due to the social, economic, and educational differences between the two countries. ³⁹ Thus, the current data will allow clinicians to make direct comparisons between scores on individual tests of the MACFIMS in reference to a more consistent and demographically relevant population of healthy controls than that afforded by previously available norms and by published test norms (which, again, are largely American). A case example of the application of regression-based normative data is presented in Text Box 1. Age, education, and sex predicted 0 to 18% of the variance in MACFIMS scores, with a mean of 7%. Age and education had the most frequent impact. The impact of demographics is small but relatively consistent with that noted in other regression-based normative data. The American regression-based MACFIMS normative study found that age, education, and sex predicted 7-28% of the variance, with age having the most frequent impact. ³⁷ Similar data for the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) in older adults revealed that age, education, sex, and race predicted 5-28% of the variance in scores. ⁴⁷ Demographics explained from about 10 to 30% of the variance in scores on the Halstead–Reitan battery. ^{48 , 49} Thus, the minimal but variable range in demographic influence observed currently is consistent with past literature.

Text Box 1 Case example of the application of regression-based normative data

Presume that one has assessed a 43-year-old woman with secondary-progressive MS who has 14 years of education. She has obtained a raw score of 21 on the BVMT–R Total Recall index. Normative data from the published BVMT–R manual yields a z score of –0.82. Using the current discrete norms for the 36 to 50 age group, her z score would be –1.39. Using the regression equation in Table 3, her demographically corrected predicted score would be 26.49 (i.e., 29.21 – [sex × 0.30] – [age × 0.16] + [educ× 0.34]=29.21 – [2 × 0.30] – [43 × 0.16] + [14 x 0.34]=26.49). The predicted score is subtracted from the observed score, and the result is divided by the standard error of the estimate for that subtest (i.e., [21 – 26.49]/5.54). This yields a z score of –0.99. As expected, given findings with the Wechsler scales, ³⁹ Canadian raw scores on the BVMT–R are generally slightly higher than American normative data. Thus, when using the current Canadian normative data to calculate a standard score, this patient is more severely impaired than the American data would suggest. Although whether the current discrete or regression-based norms are harsher depends on the task, both forms of the Canadian norms suggest more impaired functioning for this particular individual on other tasks as well (e.g., SDMT: published z=–2.05; discrete z=–2.61; regression z=–4.19; FAS: published z=–2.0; discrete z=–2.10; regression z=–3.94). http://psychology.wikia.com/wiki/Bell_curve_grading

As can be expected, the influence of particular demographic variables varied between MACFIMS tests. Tasks measuring processing speed and working memory were affected to varying degrees by demographics. SDMT performance was influenced mostly by age, whereas PASAT performance was more significantly impacted by education. The age effect on SDMT performance has been demonstrated in the literature, ⁵⁰ and the lack of age effect on the PASAT is not unexpected given that the literature demonstrates a more inconsistent influence of age. ⁵¹ Education has been demonstrated to impact PASAT performance in healthy individuals (traditional scoring) ⁵² and in people with MS (as mediated by mathematical ability). ⁵³ The latter is contributing to the growing literature demonstrating that the PASAT is not likely an appropriate instrument for monitoring cognition in MS despite the entrenchment of this measurement tool in the MS literature. ^{54 , 55} The lack of demographic influence on the COWAT is somewhat surprising, given that the literature demonstrates a general decline in performance with age and better scores in more highly educated individuals. ²⁵ The lack of age effects in our 18-to-65-year-old sample may be due to the fact that the greatest influence of age is seen at the extremes (i.e., childhood and old age). ²⁵ The lack of influence of education in our sample is likely due to the fact that the vast majority of our subjects had a high school education or above. The JLO was most influenced by age and sex, as has also been demonstrated in the literature. ¹⁰ The verbal memory test (CVLT–II) was most influenced by age and education, consistent with the findings of the test authors. ¹¹ Nonetheless, the current results did not yield the female advantage noted by the test authors, ¹¹ perhaps due to the fact that the current gender distribution was biased toward females. Performance on the nonverbal memory test (BVMT–R) was similar to that on the verbal task, with age and education being most contributory. The test author reported that age is moderately correlated with performance, whereas the relationship of performance to education is generally weak. ⁵⁶ Finally, the Sorting Test was influenced most significantly by sex. Little information is available on demographic influences on the D–KEFS measures, with the manual reporting only age effects, ¹³ so that it is unclear whether sex is also a variable that typically influences performance.

One limitation of our study is that some of the tasks (i.e., the CVLT–II and JLO) had small sample sizes (n=57) in the regression-based formulae. This value falls below the 100 participants recommended to obtain precise classifications of performance for individuals using regression-based normative data and suggests that, for these tasks in particular, caution must be used when interpreting the performance of individuals who fall at the extremes of the demographic variables of interest. ³⁰ Indeed, it has been suggested that when numbers are smaller the norms can lead to higher rates of false negatives (or missed diagnoses), particularly in older adults and those with lower levels of education. ⁵⁷ Sample size was even smaller for these same tasks in the discrete norms (e.g., as low as 16 participants in the 36-50 age group), so that, again, caution must be applied when interpreting performance. In particular, the current findings should not be utilized in individuals over the age of 65. Similarly, care should be taken when interpreting the performance of those with less than a high school education, as all but two individuals in this database had a high school education or greater. Notably, the Conference Board of Canada reports that, as of March of 2013, 88% of working-age Canadians have graduated from high school. As such, the educational background of the current sample is fairly representative of the general population. Further limitations include a lack of ethnicity data for the full sample and a lack of information on the percentage of bilingual or multilingual speakers. Although some may consider the larger proportion of females represented in this sample as a limitation, it is important to note that the data will largely be used to interpret the performance of individuals with MS. Given that the majority of people living with MS are female, the current demographic distribution is considered appropriate.

In conclusion, provision of these Canadian regression-based formulae will allow for more accurate interpretation of MACFIMS scores by allowing clinicians to correct for relevant demographic variables simultaneously, leading to improved clinical decision making for Canadian individuals with multiple sclerosis. The provision of both discrete and regression-based options allows clinicians the freedom to choose the scoring method best suited to their own practice. Our group plans to follow up on this preliminary work by more thoroughly evaluating the utility of these Canadian regression-based normative data compared to the American data in interpreting the performance of Canadian individuals with MS.