Can neuropsychological testing facilitate differential diagnosis between at-risk mental state (ARMS) for psychosis and adult attention-deficit/hyperactivity disorder (ADHD)?

Abstract

Background:

Patients with an at-risk mental state (ARMS) for psychosis and patients with attention-deficit/hyperactivity disorder (ADHD) have many overlapping signs and symptoms and hence can be difficult to differentiate clinically. The aim of this study was to investigate whether the differential diagnosis between ARMS and adult ADHD could be improved by neuropsychological testing.

Methods:

168 ARMS patients, 123 adult ADHD patients and 109 healthy controls (HC) were recruited via specialized clinics of the University of Basel Psychiatric Hospital. Sustained attention and impulsivity were tested with the Continuous Performance Test, verbal learning and memory with the California Verbal Learning Test, and problem solving abilities with the Tower of Hanoi Task. Group differences in neuropsychological performance were analyzed using generalized linear models. Furthermore, to investigate whether adult ADHD and ARMS can be correctly classified based on the pattern of cognitive deficits, machine learning (i.e. random forests) was applied.

Results:

Compared to HC, both patient groups showed deficits in attention and impulsivity and verbal learning and memory. However, in adult ADHD patients the deficits were comparatively larger. Accordingly, a machine learning model predicted group membership based on the individual neurocognitive performance profile with good accuracy (AUC = 0.82).

Conclusions:

Our results are in line with current meta-analyses reporting that impairments in the domains of attention and verbal learning are of medium effect size in adult ADHD and of small effect size in ARMS patients and suggest that measures of these domains can be exploited to improve the differential diagnosis between adult ADHD and ARMS patients.

1. Introduction

Young adults seeking help at psychiatric services frequently suffer from adult attention-deficit/hyperactivity disorder (ADHD) or an at-risk mental state (ARMS) for psychosis. Although the latter is not yet accepted as an official diagnosis, it is increasingly identified and treated in clinical practice for the following reasons. First, it is now well established that patients who meet ARMS criteria not only have a largely increased risk of developing psychosis, but also suffer from psychopathological symptoms and impaired psychosocial functioning requiring clinical attention [1]. Second, although comorbidity with other psychiatric disorders, particularly with affective and anxiety disorders, is high [2], the specific psychopathology of ARMS patients is not adequately addressed by existing diagnostic categories [3, 4]. Consequently, the European Psychiatric Association (EPA) has recently issued evidence based recommendations for the early detection [5] and treatment [6] of these patients. Furthermore, attenuated psychosis syndrome, which was defined according to the most frequent ARMS criterion, has been placed in Section 3 of the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5) [7] as a new disorder for further study.

The distinction between adult ADHD and ARMS for psychosis can be a challenging task for clinicians because these disorders have many overlapping signs and symptoms, such as difficulties concentrating, lack of attention and other cognitive deficits [8, 9], disorganized behavior, performance problems in school or at work, and problems with relationships [10, 11]. Additionally, both disorders can present with restlessness, nervousness, irritability, hypersensitivity, sudden lack of interest, initiative, energy and drive, low frustration tolerance, as well as poor resilience to stress [12, 13]. Furthermore, both disorders frequently occur in early adulthood and both are considered neurodevelopmental in origin [14, 15]. Moreover, ADHD in childhood has been found to be associated with an increased risk of schizophrenia in adulthood both in prospective follow-up [16, 17] and retrospective studies [18, 19]. Likewise, individuals genetically at-risk for schizophrenia [20, 21] and patients with psychosis [22] have been found to more frequently demonstrate ADHD-like features than healthy controls. It has been suggested that the overlap between ADHD and psychotic disorders is due to shared genetic [23] and environmental risk factors, particularly obstetric complications [24]. Another potential explanation is that – due to their similar clinical manifestation − ADHD is a frequent misdiagnosis of early signs of psychotic disorders [17].

Missclassifying patients with an ARMS for psychosis as ADHD patients can lead to inappropriate and even potentially harmful treatment of these patients. Specifically, since ADHD is most commonly treated with stimulant drugs which exert their pharmacological effects via increasing the levels of dopamine [25], and since increased levels of synaptic dopamine are implicated in the generation of psychotic symptoms [26], treating ARMS patients as ADHD patients could potentially exacerbate (pre-)psychotic symptoms in these patients [for review, see Ref. 27].

A possible way to improve differential diagnosis between these two disorders is to take the neuropsychological performance profile into account. Although current meta-analyses indicate that − compared to healthy controls (HC) – both adult ADHD and ARMS for psychosis patients show impairments across a wide range of cognitive domains, the degree and pattern of impairments seem to differ. While adult ADHD patients were found to be most strongly impaired in sustained and selective attention, inhibition, and verbal learning and memory with effect sizes in the small to medium range [9, 28, 29]. ARMS patients showed the largest impairments in speed of processing, social cognition and verbal learning with, however, mostly small effect sizes [8].

However, to our knowledge, no study has directly tested cognitive performance differences between adult ADHD and ARMS patients. Furthermore, it is currently unknown whether potential differences on the group level could be exploited to facilitate diagnostic classification on the individual level by means of automated pattern recognition or machine learning methods. This is unfortunate because neuropsychological testing is already routinely conducted in early detection services for both disorders and has shown promise in the classification between ADHD and various other psychiatric disorders, including mood and anxiety disorders [30–33], autism spectrum disorders [34], and borderline personality disorders [35], and between ARMS for psychosis and depressive disorders [36].

Thus, the aim of this study was 1) to directly test neuropsychological performance differences between adult ADHD, ARMS for psychosis patients, and HC in the domains of sustained attention and impulsivity, verbal learning and memory, and problem solving abilities and 2) to estimate the classification accuracy of machine learning model predicting group membership from all neurocognitive performance measures combined.

Based on the above mentioned meta-analyses [8, 9, 28, 29], we hypothesized that both patient groups would show worse cognitive performance in all tested domains relative to healthy controls and that adult ADHD patients would show larger deficits than ARMS patients in the domains of attention and impulsivity and verbal learning and memory, but not problem solving abilities. Furthermore, we expected that classification between adult ADHD and ARMS patients based on all neuropsychological performance measures combined can be achieved with moderate to high accuracy.

2. Methods

2.1. Study design

In this cross-sectional study, the cognitive performance of three groups was compared: 1) patients with adult ADHD, 2) patients with an ARMS, and 3) healthy controls (HC). All participants provided their written informed consent. The study was approved by the by the local ethics committee (Ethikkommission der Nordwest- und Zentralschweiz) and conformed to the Declaration of Helsinki.

2.2. Recruitment of patients with adult ADHD

Patients with adult ADHD were recruited via the ADHD Special Consultations Unit of the Outpatient Department of the University of Basel Psychiatric Hospital between 2010 and 2014. All referrals to this Unit underwent an extensive ADHD screening procedure conducted by two independent experts and including a clinical interview, a self-rating and an observer-rating scale. School certificates and/or reports from teachers on behavioral problems were also considered. The procedure conformed to general standards for clinical diagnostics and followed the recommendations for the diagnostics and management of ADHD [37]. The diagnosis was not made solely on the basis of rating scales, but it also took into account a full developmental and psychiatric history. Instruments applied in the diagnostic process were the Wender-Reimherr Adult Attention Deficit Disorder Rating Scale (WRAADDS) [38] (German version: [39, 40]) and the Conners’ Adult ADHD Rating Scales (CAARS) [41]. Former symptoms in childhood were assessed systematically with the short version of the Wender Utah Rating Scale (WURS-k) [42] (German version: [43]). To check for current symptoms, a combination of the results from the clinical interview and the rating scales was used. An ADHD diagnosis was given when an individual met at least six criteria of the dimension of inattention and/or six criteria of the dimension of hyperactivity/impulsivity according to DSM-IV. Only patients who met the diagnostic criteria for ADHD and who were at least 18 years old were included into this study. Exclusion criteria were an intelligence quotient (IQ) <85, schizophrenia or other psychotic disorders, a current or most recent episode of a manic disorder or current severe major depressive disorder, acute stress disorder, or substance intoxication or withdrawal.

2.3. Recruitment of patients with an ARMS for psychosis

ARMS patients were recruited and assessed between March 1, 2000 and December 31, 2016 as part of the Basel Früherkennung von Psychosen (FePsy) study, a prospective multilevel study, which aims to improve the early detection of psychosis. A detailed description of the study design can be found elsewhere [44, 45]. In brief, individuals suspected to be in their early (prodromal) phase of psychosis were referred to our specialized early detection clinic at the Outpatient Department of the Psychiatric University Hospital Basel, Switzerland. All referrals to the clinic during the study period were screened with the Basel Screening Instrument for Psychosis (BSIP) [46], which has been specifically designed to identify patients with an ARMS for psychosis or first episode of psychosis (FEP). Individuals were classified as being in an ARMS if they met one of the following risk criteria: (a) attenuated psychotic symptoms (APS) or brief limited intermittent psychotic symptoms (BLIPS) according to the PACE criteria [47], (b) familial aggregation of psychotic disorders in combination with at least two further risk factors similar to the PACE criteria, or (c) a minimal amount and combination of certain risk factors according to the BSIP [for details, see Ref. 46]. The BSIP has been shown to have a good interrater reliability (κ = 0.67) for the assessment of the ARMS and a high predictive validity [46]. Exclusion criteria were: age <18 years, insufficient knowledge of German, IQ below 85, previous episode of schizophrenic psychosis, psychosis clearly due to organic reasons or substance abuse, or psychotic symptomatology within a clearly diagnosed affective psychosis or borderline personality disorder. Patients who were treated with antipsychotics >3 weeks or who had exceeded a 2500 mg cumulative chlorpromazine equivalent dose were also excluded.

2.4. Recruitment of healthy controls (HC)

HC were also recruited and assessed as part of the FePsy study. They were recruited from trade schools, hospital staff and through advertisements. Subjects were excluded if they had a current or former psychiatric disorder or neurological disease, serious medical condition, substance abuse, or a family history of psychiatric disorder.

2.5. Neuropsychological assessment

Patients with adult ADHD were assessed at the ADHD Special Consultations Unit and patients with an ARMS for psychosis and HC were assessed at the early detection of psychosis clinic with neuropsychological test batteries that were overlapping in the following tests: 1) Continuous Performance Test (CPT) [48], 2) California Verbal Learning Test (CVLT) [49] and 3) the computer administered Tower of Hanoi (ToH) [50]. A detailed description of these tests is provided in the online Supplementary material.

2.6. Statistical analysis

All data were analyzed using the R language and environment for statistical computing [51]. For testing differences between adult ADHD, ARMS and HC groups on sociodemographic variables, Pearson’s χ²-tests and analysis of variance (ANOVA) were used for categorical and continuous variables, respectively.

Group differences in neuropsychological performance measures were analyzed using generalized linear models (GLM). GLMs are flexible generalizations of the ordinary linear regression, which differ from them in two major aspects: First, the distribution of the response variable can be non-normal and/or non-continuous (e.g., it can be binary, ordered categorical, a count etc.). Second, the dependent variable values are predicted from a linear combination of predictor variables, which are connected to the dependent variable via a link function (e.g. logit, log, inverse etc.). Thus, GLMs can take advantage of the specific structure of each variable and respect natural boundaries and distributions of variables when making predictions [for applied examples of GLMs with neuropsychological data, see Refs. 52, 53].

For the number of omissions and commissions in the CPT, we used negative binomial models with log-link function because these are count variables (i.e. only non-negative integer values are possible). We used negative binomial instead of standard Poisson regression for these measures because it allowed handling overdispersion [54]. For CVLT performance measures, we used binomial models with logit link function, because each measure quantified the number of words that were remembered from one or several lists of words with fixed length. For the reaction time in the CPT and the number of moves and time to complete in the ToH, Gamma regression models with an inverse link function were a natural choice because these variables were non-negative and right-skewed.

To control for confounding effects, all GLMs included sex and age as covariates. We first tested for overall group differences using likelihood ratio tests and then performed all possible pairwise comparisons using the ghlt function in the R multcomp package [55]. P-values were adjusted for multiple testing using the Benjamini-Hochberg procedure [56] both across all pairwise group comparisons and neurocognitive variables.

To find out whether all neurocognitive performance measures combined can successfully discriminate between ARMS and ADHD patients on an individual level, we additionally analyzed our data with a supervised machine learning method. Specifically, a random forest model was trained and tested that predicted group membership based on the individual performance across all neurocognitive variables. We chose the random forest algorithm because it can automatically 1) accommodate non-linear relationships between predictor and outcome variables, 2) capture complex interactions between predictors and 3) deal with highly correlated predictors, outliers and missing data [57]. Furthermore, it has been shown to achieve excellent predictive accuracy over a wide of range predictive tasks even without hyperparameter tuning [58]. To eliminate the influence of age and sex, we first created two equally sized samples of ADHD and ARMS patients that were matched on age and sex using propensity score matching as implemented in the matching package for R [59]. Next, a random forest model for binary outcomes was trained and tested using 10-fold cross validation with 10 repetitions. The discriminative ability of the model was assessed with the cross-validated area under the receiver operating characteristic curve (AUC), as well as sensitivity and specificity. Model fitting and performance evaluation was conducted with the R packages randomForest [60] and mlr [61], respectively. We additionally assessed the predictive importance of each variable by looking at how much the accuracy decreased when the variable was randomly permuted using the importance function in package randomForest [60].

3. Results

3.1. Sample description

During the above described study periods, 123 adult ADHD patients, 168 ARMS patients, and 109 HC fulfilled inclusion criteria and had completed at least one of the above described neuropsychological tests. Eleven HC and 23 ARMS patients had to be excluded from the current study due to completely missing neuropsychological assessment. The excluded patients did not differ from the included patients with regard to sociodemographic characteristics. The ADHD, ARMS and HC groups did not differ with regard to gender ratios. However, adult ADHD patients were significantly older than ARMS patients and HC (see Table 1 for socio-demographic sample characteristics).

Table 1 Socio-demographic sample characteristics.

HC = healthy controls; ARMS = at-risk mental state for psychosis; ADHD = attention-deficit/hyperactivity disorder.

3.2. Group differences in neuropsychological performance

Standardized mean differences in neuropsychological performance from HC for the ARMS and ADHD groups are displayed in Fig. 1. Means and SDs, as well as p-values for tests of overall and pairwise group differences are provided in Table 2.

Fig 1. Standardized mean differences of cognitive performance of ARMS and ADHD patients compared to healthy controls. Variables with a minus sign were reversed so that higher scores always represent better performance. Differences are adjusted for the influence of age and gender.

Table 2 Cognitive performance of ADHD and ARMS patients

CVLT = California Verbal Learning Task; CPT = Continuous Performance Task; ToH = Tower of Hanoi; HC = Healthy controls; ADHD = Attention-Deficit/Hyperactivity Disorder; ARMS = At-risk mental state for psychosis

All group comparisons are adjusted for the influence of sex and age and corrected for multiple testing using the Benjamini-Hochberg procedure

* p < 0.05.

** p < 0.01.

*** p < 0.001.

When adjusted for the influence age and sex and corrected for multiple comparisons, there were overall group differences in all tested neuropsychological variables, except for the number of moves in the 5 disc ToH. Pairwise group comparisons revealed that ADHD patients showed a significantly worse performance than HC in all CPT and CVLT measures, but not in any ToH measure. ARMS patients also performed worse than HC in all performance scores, except for List 1, Trial 1 recall and Long Delay False Alarms in the CVLT and number of moves in the 5 disc ToH. Finally, ADHD patients performed significantly worse than ARMS patients in all measures except the two ToH measures. Repeating the analyses without covariate adjustment did not change results except that ARMS patients additionally showed a significantly worse performance than HC in List 1, Trial 1 recall of the CVLT (p = 0.049). As can be seen in Fig. 1, cognitive impairments were mostly of medium effect sizes in adult ADHD patients and of small effect sizes in ARMS patients (see Fig. 1).

3.3. Discrimination between ARMS and ADHD patients using machine learning

The propensity score matching procedure selected two subgroups of ADHD and ARMS each consisting of 101 participants. The two subgroups were no longer significantly different with regard to age (ADHD: 29.0 ± 8.2 years, ARMS: 28.3 ± 7.9 years) and sex (ADHD: 34.7% Women; ARMS: 34.7% Women). The random forest algorithm was able to discriminate the two matched groups with relatively high accuracy based on all neurocognitive variables combined (cross-validated AUC = 0.82, balanced accuracy = 0.75, sensitivity = 0.73, and specificity = 0.77). The variable importance assessment revealed that CVLT long delay free recall, CPT reaction time, and CVLT recognition hits were the most important predictor variables for discriminating ADHD from ARMS patients (see Fig. 2).

Fig 2. Variable importance for predicting group membership (ARMS vs. ADHD) with random forest.

4. Discussion

To our knowledge, this is the first study aiming to discriminate adult ADHD from ARMS for psychosis patients based on neuropsychological performance. In line with our hypotheses, we could demonstrate that both patient groups show significantly worse cognitive performance than HC in the domains of sustained attention and impulsivity and verbal learning and memory and that adult ADHD patients show larger deficits in these domains than ARMS patients. Accordingly, we could demonstrate that a machine learning algorithm (i.e. random forest) can predict group membership (ADHD vs. ARMS) with relatively high accuracy solely based on the individual pattern of cognitive deficits.

Our findings of small to medium sized cognitive deficits in adult ADHD patients are largely consistent with the most recent meta-analyses on cognitive deficits in adult ADHD patients [9, 28, 29]. Furthermore, our results confirmed that deficits of attention in general and sustained attention in particular, as measured by the CPT, represent one of the core features of ADHD. However, in contrast to the most recent meta-analysis of the CPT [28], which reported the strongest effect sizes for the number of omissions and commissions and no statistically significant impairment with regard to reaction times, we observed significant impairments in all performance scores of the CPT, with reaction time being the most strongly impaired. This difference might be explained by the fact that the most commonly used CPT version, the Conner’s CPT [41], has 90% target trials with responses required for any letter except “X”, whereas the CPT version used in this study has only 25% target trials with responses required whenever an “O” is followed by an “X”. Our results also support a recent meta-analysis on memory performance in adult ADHD patients, which found that these patients show moderate impairments in verbal learning and memory resulting from deficits in memory acquisition, but not retrieval problems [29].

The cognitive performance deficits that we found in our sample of ARMS patients are also largely consistent with those reported in recent meta-analyses [8, 62], which have detected impairments of mostly small effects sizes in the domains of sustained attention, verbal learning and memory and executive functions.

In accordance with our hypothesis and reported effect sizes of meta-analyses on adult ADHD [9, 28, 29] and ARMS patients [8, 62], we could demonstrate that adult ADHD patients are significantly more impaired than ARMS patients in the domains sustained attention and impulsivity and verbal learning and memory, but not in problem solving abilities. Furthermore, we could show that these differences on the group level can be exploited to make meaningful prediction on the individual level. Specifically, by applying a machine learning algorithm to all neurocognitive variables combined we were able to discriminate ADHD from ARMS patients with a cross-validated sensitivity of 0.73 and specificity of 0.77. Thus, our results clearly support the notion that neuropsychological testing could facilitate the differential diagnosis between these two disorders. However, it should be noted that our results do not indicate that the pattern of cognitive deficits alone could predict the correct diagnosis with sufficient certainty, only that it could provide important hints when combined with other signs and symptoms.

4.1. Strengths and limitations

A limitation of this study is that adult ADHD and ARMS patients were recruited from two different psychiatric services, albeit in the same psychiatric university hospital and across a similar time period. Thus, differences in neuropsychological performance could only be tested in the three cognitive tasks that were overlapping across the two services. Consequently, other important cognitive domains, such as figural memory, verbal fluency, speed of processing, and social cognition could not be investigated. Another limitation is that ADHD patients were not assessed with the BSIP for being in an ARMS for psychosis. Hence, we cannot rule out the possibility that at least some patients would fulfill criteria for both disorders. Lastly, the generalization of our results is limited by the fact that women and patients diagnosed with the inattentive subtype of ADHD were underrepresented in our sample of adult ADHD patients.

Strengths of our study are the relatively large sample sizes and the fact that relatively few patients in both patient groups were treated with medication (see Table 1). Thus, only few ADHD patients were treated with methylphenidate, which has been shown to improve performance in the CPT and other cognitive tasks in adult ADHD patients [63]. Likewise, only few ARMS and adult ADHD patients were treated with antipsychotics, which have been shown to negatively affect cognition, particularly speed of processing [64].

4.2 Conclusions

The results of this study suggest that neuropsychological testing, particularly of the domains of sustained attention and impulsivity and verbal learning and memory, can help to improve the differential diagnosis between adult ADHD and ARMS for psychosis. Future studies should replicate and extend our findings with larger samples, and more comprehensive neuropsychological test batteries.

Declarations of interest

None.