24 Demographic Adjustment Is Not Demographic Correction: A Simulation Study

Abstract

Objective:

Prior studies have presented demographic adjustment as beneficial because it helps equalize, across demographic groups, the percentage of participants (recruited from the general population without prior diagnosis) who fell beneath the test impairment cutoff (e.g., Smith, et al., 2008). This methodology ignores the possibility that group differences in those falling beneath an impairment cutoff could reflect cognitive impairment prevalence differences between demographic groups in the undiagnosed general population. Demographic group differences in cognitive test scores reflect a mixture of two categories of influences: measurement bias (item/test/examiner bias, language/cultural bias, stereotype threat, etc.) and factors which differentially increase the number of low scores in one group by increasing relative risk (RR) for cognitive impairment (biological aging processes, cognitive reserve, social determinants of health [SDoH], etc.). The current simulation study examined how the effect of demographic adjustment on the diagnostic accuracy of a hypothetical test (operationalized as the area under the curve [AUC] in an ROC analysis) varied as the mixture of influences which caused demographic differences in scores were varied.

Participants and Methods:

215,040 samples were randomly generated. Each sample consisted of two demographic groups, with Group 0 always representing the lower scoring group. Across samples, Group 1's baseline risk of impairment and Group 0's relative risk were varied, and these determined the prevalence of cognitive impairment in the groups. Three facets of measurement bias were varied in the simulation: how much lower Group 0's average score was than Group 1's, the degree of non-homogeneity of variance between groups, and how much less reliable the measure was for Group 0. Additional parameters were included and varied to ensure the robustness of findings across a variety of situations. Samples reflected all possible combinations of all varied parameters. For each sample, a baseline AUC was calculated when impairment was regressed on the unadjusted test score. Then, test scores were adjusted for demographic group and difference in adjusted and unadjusted AUC was calculated. This adjusted/unadjusted AUC difference was then regressed on the simulation parameters to quantify their relative influence.

Results:

The more Group 0's average score was reduced by measurement bias, the more improvement in AUC was seen after adjustment (ß = 1.76). Trivial but significant main effects of variance non-homogeneity (ß = .09), increased relative risk (ß = -.08), and reduced reliability (ß = .02) were also found, but more importantly, each of these predictors significantly interacted with Group 0 mean score reduction, such that higher relative risks (ß = -1.22), lower reliability (ß = .36), and higher variance (ß = -.15) in Group 0 compared to Group 1 each reduced the association between Group 0 mean score reduction and improvement in AUC.

Conclusions:

Demographic adjustment only improves AUC when the mean reduction in scores due to measurement bias is sufficiently high while risk for impairment, test reliability and test score variances are sufficiently equivalent among the demographic groups. When this is not the case, demographic adjustment can be counter-productive, reducing the AUC of the test. We conclude by proposing a novel method for adjusting test scores.