In 1984 Jacobson and colleagues introduced the concept of reliable change, viz the amount of change on a measure that an individual needed to show to determine that it exceeded the extent of change likely due to measurement error alone. Establishing reliable change was a pre-requisite for determining clinical significance. This paper summarizes the rationale for determining reliable change as providing an individual-focused, idiographic alternative to the dominant nomothetic approach to clinical outcome research based on group mean data and statistical significance. The conventional computational steps for calculating an individual’s standardized difference (reliable change) score and the minimum raw change score on the measure (a reliable change index) required to classify individuals as reliably positively changed, indeterminate, or reliably deteriorated are described. Two methods for graphically representing reliable change are presented, and a range of possible uses in both research and practice settings are summarized. A number of issues and debates concerning the calculation of reliable change are reviewed. It is concluded that the concept of reliable change remains useful for both cognitive behavioural researchers and practitioners, but that there are options regarding methods of computation. In any use of reliable change, the rationale for selecting among method options and the exact computations used need clear and careful description so that we can continuously judge the utility and appropriateness of the use of reliable change and enhance its value to the field.

Objectives: To describe multivariate base rates (MBRs) of low scores and reliable change (decline) scores on Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) in college athletes at baseline, as well as to assess MBR differences among demographic and medical history subpopulations. Methods: Data were reported on 15,909 participants (46.5% female) from the NCAA/DoD CARE Consortium. MBRs of ImPACT composite scores were derived using published CARE normative data and reliability metrics. MBRs of sex-corrected low scores were reported at <25th percentile (Low Average), <10th percentile (Borderline), and ≤2nd percentile (Impaired). MBRs of reliable decline scores were reported at the 75%, 90%, 95%, and 99% confidence intervals. We analyzed subgroups by sex, race, attention-deficit/hyperactivity disorder and/or learning disability (ADHD/LD), anxiety/depression, and concussion history using chi-square analyses. Results: Base rates of low scores and reliable decline scores on individual composites approximated the normative distribution. Athletes obtained ≥1 low score with frequencies of 63.4% (Low Average), 32.0% (Borderline), and 9.1% (Impaired). Athletes obtained ≥1 reliable decline score with frequencies of 66.8%, 32.2%, 18%, and 3.8%, respectively. Comparatively few athletes had low scores or reliable decline on ≥2 composite scores. Black/African American athletes and athletes with ADHD/LD had higher rates of low scores, while greater concussion history was associated with lower MBRs (p < .01). MBRs of reliable decline were not associated with demographic or medical factors. Conclusions: Clinical interpretation of low scores and reliable decline on ImPACT depends on the strictness of the low score cutoff, the reliable change criterion, and the number of scores exceeding these cutoffs. Race and ADHD influence the frequency of low scores at all cutoffs cross-sectionally.

Objective: Obtaining baseline neuropsychological (NP) data to assist management of sports-related concussion has been considered the standard of care. The validity of this approach has been questioned, with suggestions that post-concussion testing alone will suffice. The present study compared the sensitivity of baseline and normative paradigms in the setting of sports-related concussion.

Method: Baseline NP data were collected for 194 Australian rugby league athletes on a brief battery of paper-and-pencil NP tests. During competition, 27 athletes sustaining concussion referred from a sports physician were retested within two days of injury. Twenty-six uninjured controls were assessed at similar intervals. The baseline paradigm was assessed using a reliable change index for pre- and post-concussion scores. The normative paradigm was assessed comparing the post-concussion score to a normative mean.

Results: The baseline paradigm was consistently more sensitive to negative change following concussion than the normative paradigm when using continuous data, despite reasonable agreement. However, when data were categorised as ‘impaired’ or ‘not-impaired’, using either 68% or 90% confidence intervals, the difference between paradigms failed to reach significance. Comparison of ROC curves for both paradigms found superior overall classification for one test and the composite score using baseline comparison data.

Conclusions: Despite being a time and resource intensive process, the baseline paradigm as a repeated-measures design may be more sensitive than the between-subjects design of the normative paradigm for detecting changes following concussion. Further work is required to determine the validity of normative assessment in sports-related concussion.

Two major challenges facing researchers interested in cognitive change are that measures of change are often not very reliable, and they may reflect effects of prior test experience in addition to the factors of primary interest. One approach to dealing with these problems is to obtain multiple measures of change on parallel versions of the same tests in a measurement burst design. A total of 783 adults performed three parallel versions of cognitive tests on two occasions separated by an average of 2.6 years. Performance increased substantially across the three sessions within each occasion, and for all but vocabulary ability these within-occasion improvements were considerably larger than the between-occasion changes. Reliabilities of the changes in composite scores were low, but averages of the three changes had larger, albeit still quite modest, reliabilities. In some cognitive abilities individual differences were evident in the relation of prior test experience and the magnitude of longitudinal change. Although multiple assessments are more time consuming than traditional measurement procedures, the resulting estimates of change are more robust than those from conventional methods, and also allow the influence of practice on change to be systematically investigated. (JINS, 2012, 18, 1–8)

This study investigated the specific patient factors that predict responsiveness to a cognitive rehabilitation program. The program has previously been demonstrated to be successful at the group level in patients with gliomas, but it is unclear which patient characteristics optimized the effect of the intervention at the individual level. Four categories of possible predictors of improvement were selected for evaluation: sociodemographic and clinical variables, self-reported cognitive symptoms, and objective neuropsychological test performance. Hierarchical logistic regression analyses were conducted, beginning with the most accessible (sociodemographic) variables and ending with the most difficult (baseline neuropsychological) to identify in clinical practice. Nearly 60% of the participants of the intervention were classified as reliably improved. Reliable improvement was predicted by age (p = .003) and education (p = .011). Additional results suggested that younger patients were more likely to benefit specifically from the cognitive rehabilitation program (p = .001), and that higher education was also associated with improvement in the control group (p = .024). The findings are discussed in light of brain reserve theory. A practical implication is that cognitive rehabilitation programs should take the patients’ age into account and, if possible, adapt programs to increase the likelihood of improvement among older participants. (JINS, 2011, 17, 256–266)

Traditional methods for assessing the neurocognitive effects of epilepsy surgery are confounded by practice effects, test-retest reliability issues, and regression to the mean. This study employs 2 methods for assessing individual change that allow direct comparison of changes across both individuals and test measures. Fifty-one medically intractable epilepsy patients completed a comprehensive neuropsychological battery twice, approximately 8 months apart, prior to any invasive monitoring or surgical intervention. First, a Reliable Change (RC) index score was computed for each test score to take into account the reliability of that measure, and a cutoff score was empirically derived to establish the limits of statistically reliable change. These indices were subsequently adjusted for expected practice effects. The second approach used a regression technique to establish “change norms” along a common metric that models both expected practice effects and regression to the mean. The RC index scores provide the clinician with a statistical means of determining whether a patient's retest performance is “significantly” changed from baseline. The regression norms for change allow the clinician to evaluate the magnitude of a given patient's change on 1 or more variables along a common metric that takes into account the reliability and stability of each test measure. Case data illustrate how these methods provide an empirically grounded means for evaluating neurocognitive outcomes following medical interventions such as epilepsy surgery. (JINS, 1996, 2, 556–564.)

Different authors have used different estimates of variability in the denominator of the Reliable Change Index (RCI). Maassen attempts to clarify some of the differences and the assumptions underlying them. In particular he compares the ‘classical’ approach using an estimate SEd supposedly based on measurement error alone with an estimate SDiff based on the variability of observed differences in a population that should have no true change. Maassen concludes that not only is SEd based on classical theory, but it properly estimates variability due to measurement error and practice effect while SDiff overestimates variability by accounting twice for the variability due to practice. Simulations show Maassen to be wrong on both accounts. With an error rate nominally set to 10%, RCI estimates using SDiff wrongly declare change in 10.4% and 9.4% of simulated cases without true change while estimates using SEd wrongly declare change in 17.5% and 12.3% of the simulated cases (p < .000000001 and p < .008, respectively). In the simulation that separates measurement error and practice effects, SEd estimates the variability of change due to measurement error to be .34, when the true variability due to measurement error was .014. Neuropsychologists should not use SEd in the denominator of the RCI. (JINS, 2004, 10, 899–901.)

Researchers and clinicians using Jacobson and Truax's index to assess the reliability of change in patients, or its counterpart by Chelune et al., which takes practice effects into account, are confused by the different ways of calculating the standard error encountered in the literature (see the discussion started in this journal by Hinton-Bayre). This article compares the characteristics of (1) the standard error used by Jacobson and Truax, (2) the standard error of difference scores used by Temkin et al. and (3) an adaptation of Jacobson and Truax's approach that accounts for difference between initial and final variance. It is theoretically demonstrated that the last variant is preferable, which is corroborated by real data. (JINS, 2004, 10, 888–893.)