Long-term validity of the At Risk Mental State (ARMS) for predicting psychotic and non-psychotic mental disorders

Abstract

Background

The long-term clinical validity of the At Risk Mental State (ARMS) for the prediction of non-psychotic mental disorders is unknown.

Methods

Clinical register-based cohort study including all non-psychotic individuals assessed by the Outreach And Support in South London (OASIS) service (2002–2015). The primary outcome was risk of developing any mental disorder (psychotic or non-psychotic). Analyses included Cox proportional hazard models, Kaplan–Meier survival/failure function and C statistics.

Results

A total of 710 subjects were included. A total of 411 subjects were at risk (ARMS+) and 299 not at risk (ARMS−). Relative to ARMS−, the ARMS+ was associated with an increased risk (HR = 4.825) of developing psychotic disorders, and a reduced risk (HR = 0.545) of developing non-psychotic disorders (mainly personality disorders). At 6-year, the ARMS designation retained high sensitivity (0.873) but only modest specificity (0.456) for the prediction of psychosis onset (AUC 0.68). The brief and limited intermittent psychotic symptoms (BLIPS) subgroup had a higher risk of developing psychosis, and a lower risk of developing non-psychotic disorders as compared to the attenuated psychotic symptoms (APS) subgroup (P < 0.001).

Conclusions

In the long-term, the ARMS specifically predicts the onset of psychotic disorders, with modest accuracy, but not of non-psychotic disorders. Individuals meeting BLIPS criteria have distinct clinical outcomes.

Significant outcomes

In the long-term, the ARMS designation is still significantly associated with an increased risk of developing psychotic disorders but its prognostic accuracy is only modest. There is no evidence that the ARMS is associated with an increased risk of developing non-psychotic mental disorders. The BLIPS subgroup at lower risk of developing non-psychotic disorders compared to the APS subgroup.

Limitations

While incident diagnoses employed in this study are high in ecological validity they have not been subjected to formal validation with research-based criteria.

1. Introduction

The At Risk Mental State (ARMS) construct was introduced two decades ago, in 1996 [1], to allow identification of subjects at clinical high-risk for psychosis before full symptoms manifest. Subjects with suspected psychosis risk are usually referred to specialized services, where they undergo a specific psychometric assessment, such as the Comprehensive Assessment of At Risk Mental State (CAARMS) [1]. Upon completion of this assessment by expert and trained clinicians, referred subjects are assigned a status of being at risk (ARMS+) or not at risk (ARMS−) for psychosis [2]. Focused interventions are offered to those deemed ARMS+, in the light of their enhanced risk of developing psychosis [3]. Conversely, ARMS− subjects are usually discharged from these services and referred to other teams or to general practitioners [4]. Since its inception, the ARMS construct has gained substantial traction to the point that specialist ARMS provision has been recognized as an important component of clinical services for early psychosis intervention [5,6] (e.g. NICE guidelines [7]; recent NHS England Access and Waiting Time [AWT] standard [5], DSM-5 diagnostic manual) [8].

The broad prognostic validity of the ARMS designation is indexed by its ability to improve the pretest risk (for details see [9]) of developing mental disorders in subjects referred to high-risk services (i.e. in those later deemed ARMS+ or ARMS−) [10]. Conducting long-term studies on ARMS+ and ARMS− cohorts can be complicated due to subject attrition, particularly when following ARMS− individuals. In fact, beyond the original validation study [11], no further large-scale studies have been designed to directly address the long-term clinical validity of the ARMS designation for psychosis prediction [2]. Our recent meta-analysis showed that the three ARMS studies that are available have reported only indirect data, including small samples [12,13] or an incomplete follow-up of the ARMS− group [14]. Furthermore, the broad long-term clinical outcomes of the ARMS− group remain unknown. As a result, the specificity of the ARMS for the prediction other non-psychotic mental disorders, is not yet fully established. Recent clinical staging models [15] have suggested that ARMS+ subjects may be also at increased risk also for the development of non-psychotic mental disorders. These concerns arise in part because most ARMS+ subjects will not develop full psychosis [16]. Whether the ARMS signposts specifically to risk for future psychosis, or to nonspecific deterioration in mental health, is of paramount relevance for both clinical and research perspectives.

The present study aims to address these gaps in the literature. We used a large, real-world sample of subjects accessing a high-risk service, with a long follow-up period, to investigate the long-term clinical validity of the ARMS assessment. We first reported the pretest risk for the development of any mental disorder, to account for the initial level of risk in this selected population. We then investigated the long-term prognostic accuracy of the ARMS designation for the prediction of both psychotic and non-psychotic disorders.

2. Methods

2.1. Sample

We included all non-psychotic subjects assessed for suspicion of psychosis risk by the Outreach and Support in South London (OASIS) high-risk service, South London and the Maudsley NHS Foundation Trust (SLaM) [17]. The OASIS is a specialised clinical service for the assessment and treatment of ARMS individuals. Established in 2001 and currently led by one of the authors of the present study (PFP) it is one of the largest services of this type in Europe. All help-seeking subjects referred to OASIS on suspicion of psychosis risk in the period 1st January 2002 to 31st December 2015 were initially considered eligible. We then excluded those who were referred but never assessed by the team, and those who were already psychotic at baseline. The remaining sample was therefore composed of all non-psychotic subjects undergoing a CAARMS-based assessment at OASIS, as part of the standard care. The details of the CAARMS assessment at OASIS are detailed in a separate paper [18]. Upon completion of the assessment, these subjects were assigned the status ARMS+ or ARMS−. Details of the specific care received at OASIS team have been described elsewhere [19].

2.2. Study measures

The primary outcome of interest was the hazard ratio (HR) of developing any ICD-10 non-organic mental disorders in ARMS+ subjects as compared to ARMS− subjects (see supplementary data, eMethod 1).

Time to diagnosis of a mental disorder was measured from the date of the ARMS assessment conducted at the OASIS, censored at 1st March 2016.

Descriptive sociodemographic variables were: age [20], gender [20], ethnicity [20] (black, white, Asian, Caribbean, mixed, other), familial environment [21] (marital status: married, divorced or separated, single, in a relationship), and socioeconomic status [22] (index of multiple deprivation, IMD 2015 [23], see supplementary data, eMethod 2). All sociodemographic variables were those recorded closest to the time of first referral to OASIS.

2.3. Procedure

Clinical register-based cohort study. Primary outcome and sociodemographic variables were automatically extracted from electronic medical records with the use of the Clinical Record Interactive Search (CRIS) tool [24] (see supplementary data, eMethod 3).

2.4. Statistical analysis

Sociodemographic characteristics of the ARMS+ vs. ARMS− samples were described by means and SDs for continuous variables, and absolute and relative frequencies for categorical variables. Baseline ARMS+ vs. ARMS−characteristics were compared using Student's t-tests and Chi². The clinical validity of the ARMS assessment was investigated with Cox proportional hazards models (non-competing risk), evaluating the effects of ARMS status (ARMS+ vs. ARMS−) on the development of any incident mental disorders (any mental disorders, psychotic disorders, non-psychotic disorders) and time to development of these disorders, after checking for proportional hazards assumption [25]. Incident disorders were defined as the emergence of an ICD-10 primary diagnosis from the aforementioned groups, at any time during the follow-up, when no primary diagnosis in that ICD-10 group was present at baseline (in the first three months following referral to OASIS). We also described the impact of ARMS+ subgroups (Attenuated Psychotic Symptoms [APS]; Genetic Risk and Deterioration [GRD]; Brief and Limited Intermittent Psychotic Symptoms [BLIPS]) vs. ARMS− on the development of incident mental disorders, psychotic disorders and non-psychotic disorders. Subjects meeting multiple ARMS criteria were stratified for symptom severity as previously suggested [16]: any BLIPS > APS or APS + GRD > GRD alone. We further described the cumulative incidence of the outcome of interest with Kaplan–Meier failure function (1-survival) [26]. Clinical validity (apparent performance) was determined with the C statistic (area under the curve [AUC]). All analyses were conducted in STATA 13 (STATA Corp., TX, USA).

3. Results

3.1. Sociodemographic and clinical characteristics of the sample

From 2002 to 2015, a total of 1115 subjects were referred to the OASIS clinic for ARMS assessment. Among them, 125 subjects did not undergo the ARMS assessment and had no contact with the OASIS service. An additional 280 subjects were already psychotic at baseline (the clinical fate of these subjects is described elsewhere [27]). A final sample of 710 non-psychotic subjects who underwent ARMS assessment was used in the analyses. The sample included 411 ARMS+ subjects and 299 ARMS− subjects (Table 1). The average age of the sample was 23 years (range 12–44), with 56% males. Half of the sample was of white ethnicity, the vast majority was single and the mean IMD score was 32%. There were no significant differences in sociodemographic characteristics between ARMS+ and ARMS−, with the exception of ethnicity; there were more ARMS+ subjects of black ethnicity as compared to ARMS− subjects. The mean follow-up time was of 1472 days (SD 1171 days).

Table 1 Sociodemographic characteristics of subjects undergoing ARMS assessment at the OASIS clinic (n = 710).

ARMS: At Risk Mental State; OASIS: Outreach And Support in South London.

3.2. Pretest risk of developing any mental disorders in subjects undergoing ARMS assessment

The pretest probability of developing any mental disorder in the entire pool of subjects undergoing ARMS assessment (n = 710, Fig. 1), indicated a 6-year cumulative incidence of 0.44 (95% CI: 0.395–0.494). Since the last failure was observed at 2192 days/6.01 years, when 133 subjects were still at risk (not censored), in the following analyses we report the descriptive cumulative incidence of the failure functions at this timepoint.

Fig. 1 Cumulative incidence (Kaplan–Meier failure function) for pretest risk of developing any mental disorders in subjects undergoing At Risk Mental State (ARMS) assessment. The dotted line indicates the last event (failure) at 2192 days.

3.3. Long-term clinical validity of the At Risk Mental State

3.3.1. Prediction of any mental disorder

There were no significant between-group differences in hazard risks (HR = 0.979, Table 2). The 6-year cumulative incidence was 0.445 (95% CI: 0.387–0.507) in the ARMS+, and 0.431 (95% CI: 0.347–0.525) in the ARMS− (supplementary data, eFigure 1). The mean time to event in ARMS+ was 2979 days (95% CI: 2733–3225) and in ARMS− was 2584 (95% CI: 2299–3225).

Table 2 Long-term clinical validity of the ARMS for the prediction of mental disorders. Cox proportional hazards analyses. Failure events were defined as the emergence of an ICD-10 primary diagnosis from the different groups, at any time during the follow-up, when no primary diagnosis in that ICD-10 group was present at baseline.

ARMS: At Risk Mental State; na: not available; Se: sensitivity; Sp: specificity; AUC: area under the curve.

^a ARMS+ vs. ARMS− (base).

3.3.2. Prediction of psychotic disorders

There were significant between-group differences in hazard risk, with higher risk of psychosis in the ARMS+ as compared to the ARMS− (HR = 4.83, Table 2). The 6-year cumulative incidence was 0.201 (95% CI: 0.161–0.250) in the ARMS+ and 0.042 (95% CI: 0.022–0.076) in the ARMS− (Fig. 2). The mean time to event in ARMS+ was 4124 days (95% CI: 3933–4315) and in ARMS− was 3940 days (95% CI: 3847–4033).

Fig. 2 Cumulative incidence (Kaplan–Meier failure function) for the long-term risk of development of psychotic disorders in At Risk Mental State (ARMS)+ (n = 411) and ARMS− (n = 299) subjects. LR+ 1.612, LR− 0.276.

At the 6-year timepoint, the ARMS assessment showed a very good sensitivity (0.873, Table 2) but only a modest specificity (0.456). This was reflected by a moderate negative likelihood ratio (LR−) of 0.276 and a small positive likelihood ratio (LR+) of 1.612 (for details on LR−, LR+ and probabilistic prognostic reasoning in ARMS see [9]), with a modest AUC (mean 0.68, 95% CI: 0.62–0.70).

3.3.3. Prediction of non-psychotic disorders

There were significant between-group differences in hazard risks between the ARMS+ and ARMS− (HR = 0.545, Table 2), with higher risk of non-psychotic disorders in the ARMS− than in the ARMS+ group. The 6-year cumulative incidence was 0.281 (95% CI: 0.226–0.347) in the ARMS+ and 0.404 (95% CI: 0.320–0.501) in the ARMS− (supplementary data, eFigure 2). The mean time to event in ARMS+ was 3710 days (95% CI: 3473–3946), and in ARMS− 2691 days (95% CI: 2404–2978).

3.3.4. Prediction of specific non-psychotic disorders

There were no significant between-group differences in the hazard risks for the development of bipolar mood disorders (supplementary data, eFigure 3), non-bipolar mood disorders (supplementary data, eFigure 4), anxiety disorders (supplementary data, eFigure 5), substance use disorders, disorders with childhood/adolescence onset, or physiological syndromes (Table 2 and supplementary data, eResults). Conversely, there was higher risk of development of personality disorders (48% of the failures were coded as ICD-10 F63 emotionally unstable personality disorders) in the ARMS− as compared to the ARMS+ group (HR = 0.179, Table 2). The 6-year cumulative incidence was 0.022 (95% CI: 0.009–0.054) in the ARMS+ and 0.095 (95% CI: 0.058–0.154) in the ARMS− (supplementary data, eFigure 6). There was also higher risk of developing developmental disorders in the ARMS− as compared to the ARMS+ (HR = 0.339, Table 2) but there were only a very few (n = 2) failures. The 6-year cumulative incidence was 0.019 (95% CI: 0.004–0.090) in the ARMS−, while there were no failures in the ARMS+ subgroup.

3.3.5. ARMS subgroups and prediction of mental disorders

There were not enough cases in the GRD subgroup to allow meaningful statistical analyses (n = 6). These subjects were therefore discarded from the following analyses. When the APS and BLIPS subgroups where compared with ARMS−, there were no significant between-group differences in the risk of development of any mental disorders (P = 0.892). However, there were significant differences in the risk of developing psychotic disorders between the groups (P < 0.001). There were also significant between-group differences in the risk of developing non-psychotic mental disorders (P < 0.001), with the lowest risk in the BLIPS subgroup, the highest risk in the ARMS− subgroup and the APS subgroup in an intermediate position (Fig. 3).

Fig. 3 Cumulative incidence (Kaplan–Meier failure function) for the long-term risk of development of non-psychotic disorders in At Risk Mental State (ARMS)+ (APS) (n = 299), ARMS+ (BLIPS) (n = 62) and ARMS− (n = 228) subjects.

4. Discussion

This study has the largest sample size and longest follow-up period of any study that has investigated the real-world clinical validity of the ARMS designation. In subjects undergoing ARMS assessment, the 6-year pretest risk for the development of any mental disorder was 0.44 and higher than in unselected samples. At 6-year follow-up, the ARMS+ group was associated with a fivefold risk of developing psychosis as compared to the ARMS− group. In the long-term, the CAARMS retained very good sensitivity but only modest specificity. The ARMS+ was associated with a lower risk of developing non-psychotic disorders (mostly personality disorders) relative to the ARMS−. Among ARMS+ subgroups, the BLIPS subgroup had a lower risk of developing non-psychotic disorders than the APS subgroup.

To our knowledge, this is the first study to investigate the risk of developing any mental disorder (psychotic and non-psychotic) in subjects undergoing and completing an ARMS assessment. Understanding whether the ARMS status delineates specific risk for developing mental disorders necessarily relies upon the reporting of incident rates of different classes of psychiatric disorders. Because our database drew directly from real-world and real-time electronic health records, we were able to track the incident diagnoses of all ICD-10 non-organic mental disorders. This approach allowed us to estimate the overall burden of risk of subjects referred to high-risk services. We found that their overall pretest risk of developing any mental disorder (i.e. before completion of the ARMS assessment) accumulated to approximately 44% at 6 years. This value is higher than the 6-year incidence of 27.84% (95% CI: 27.24%–28.44% estimated from a previous study [28]) for any mental disorder in primary care settings, consistent with ARMS subjects representing selected help-seeking samples. These findings are in line with the recently observed risk enrichment and increased vulnerability for the development of mental disorders in subjects seeking help from high-risk services [10]. Because the observe pretest risk enrichment is substantial, pretest risk stratification models have been recently developed and validated by our group [29].

The CAARMS assessment then defined the ARMS+ and ARMS− groups from this selected and help-seeking population. There was no significant difference in the overall 6-year cumulative incidence of any mental disorders between ARMS+ (45%) and ARMS− (43%) (supplementary data, eFigure 1). However, there was an increased risk for psychotic disorders in ARMS+ and an increased risk for non-psychotic disorders in the ARMS−.

We have thus replicated the earlier findings of the original validation study [11] by confirming that the ARMS+ was associated with greater risk of developing psychotic disorders, even in the longer term. The ARMS assessment retained a good ability to rule out psychosis (as reflected by the high sensitivity, 0.873), but was associated with an inadequate ability to rule in psychosis (as reflected by the modest specificity, 0.456) (Fig. 2). Similarly, the LR− of 0.276 indexed a moderate [30] decrease of pretest probability for psychosis following an ARMS− designation, while the LR+ of 1.612 indexed only a slight [30] increase in pretest probability for psychosis following an ARMS+ designation. These results indicate a modest long-term prognostic accuracy (AUC 0.68) and the need to specifically improve the ability to rule in subsequent psychosis, while preserving the outstanding ability to rule it out. As the clinical gain of testing positive at an ARMS assessment is modest, it is therefore essential to use it in samples that are already risk enriched such as those accessing mental health services [31]. The use of the ARMS assessments outside clinical samples is likely to dilute the pretest risk and consequently the transition rates to psychosis [32]. Sequential testing with combinations of predictive models deriving from clinical, neurocognitive and biological domains are currently being investigated to overcome some of these caveats [33].

Conversely, the ARMS+ was not associated with a higher risk of developing mental disorders other than psychosis, relative to the ARMS− (supplementary data, eFigure 2). Although the risk for non-psychotic disorders was significantly lower in the ARMS+ relative to the ARMS−, approximately 27% of ARMS+ developed a non-psychotic disorder by the 6-year timepoint. This comes in addition to the high baseline prevalence of non-psychotic comorbid disorders [34], impaired functioning [35] and the persistence of non-psychotic comorbid disorders over follow-up [36] in ARMS+ samples. Taken together, these results do not support the notion of diagnostic pluripotentiality in the ARMS+. As a risk state specific for psychosis, the possible outcomes specifically associated with the ARMS+ designation may include onset of psychotic disorders, remission or persistence of initial ARMS symptoms and variable functional outcomes, but not an increased risk of emergence of non-psychotic mental disorders.

We also specifically investigated the type of non-psychotic disorders associated with an ARMS designation. We confirmed the findings of previous studies in clinical high-risk samples, reporting no increased risk for bipolar mood disorders, non-bipolar mood disorders or anxiety disorders [37]. Our 6-year 1.7% cumulative incidence for bipolar disorders in the ARMS+ (supplementary data, eFigure 3) matches (albeit at different timepoints) to the previous 1.9% reported in the NAPLS-1 cohort [37]. Similarly, our 6-year 11.4% cumulative incidence for anxiety disorders (supplementary data, eFigure 5) in the ARMS+ is close to the 10.8% rate reported in the PREDICT sample [37]. Conversely, our 6-year 10% cumulative incidence of non-bipolar mood disorders (supplementary data, eFigure 4) appeared higher than the rates reported in NAPLS-1 [37]. However, the comparability of these findings may be problematic due to the different time-points, study designs employed and the operational differences between the CAARMS and the Structured Interview for Psychosis-Risk Syndromes (SIPS) [38]. More importantly, the cumulative incidences of these non-psychotic disorders in the ARMS+ are similar to the annualized 6-year rates estimated from studies conducted in general community studies for bipolar disorders (0.08% estimated from [28]), non-bipolar mood disorders (8.29% estimated from [39]) and anxiety disorders (9.48% estimated from [40]). Moreover, the cumulative incidence of these disorders in the ARMS+ was lower than in population-based studies of young adults at high-risk of bipolar [41], non-bipolar mood [42] and anxiety [43] disorders. Overall, these findings suggest that the ARMS could not effectively be used as a preventative paradigm to alter the course of these non-psychotic disorders.

We have also shown, for the first time, no differences between ARMS+ and ARMS− in risk for the development of substance use disorders, disorders with childhood/adolescence onset or physiological syndromes, and uncertain findings with respect to developmental disorders (due to the rare events). Conversely, we found that the ARMS− group had an increased risk for the development of personality disorders (supplementary data, eFigure 6). The 6-year cumulative incidence of personality disorders was high in the ARMS−, at 9.5%. Unfortunately, it is not possible to compare this incidence rate with that of the general population because the latter is unknown. Studies in patients admitted to psychiatric services have reported incidence rates of ICD-10 personality disorders of 11% during a 12-year period [44] (supplementary data, eDiscussion 1). Future studies may compare risk of development of non-psychotic disorders between subjects undergoing ARMS assessment and healthy controls.

We additionally explored the impact of the type of ARMS subgroup on long-term clinical outcomes. We have previously shown that relative to the APS subgroup, the BLIPS subgroup has a greater risk of developing psychosis [16]. In previous publications, we also demonstrated that risk of developing psychosis in BLIPS cases is comparable to concurrent ICD-10 diagnoses traditionally employed to describe brief psychotic episodes [45]. The current findings provide further evidence for the distinctiveness of the BLIPS subgroup as compared to the APS [45,46]. More specifically, we found that the BLIPS were less likely to transition to non-psychotic disorders, relative to the APS. The high specificity towards psychosis, coupled with the low risk of development of non-psychotic disorders, suggest that the BLIPS subgroup is composed of psychotic subjects with an endophenotype of the disorder that is characterized by short and remitting phases and may represent a distinct clinical stage as compared to the APS subgroup [47].

The principal limitation of the current study is that we did not employ a structured psychometric interview to ascertain the type of incident diagnoses at follow-up. Therefore, while the incident diagnoses are high in ecological validity (i.e. they represent real-world clinical practice), they have not been subjected to formal validation with research-based criteria. However, as previously noted in these samples [37], the use of structured diagnostic interviews can lead to selection of patient subsamples and introduce additional biases. Furthermore, there is also meta-analytical evidence indicating that for some psychotic categories, administrative data recorded in clinical registers are generally predictive of true diagnosis [48].

5. Conclusions

Subjects meeting ARMS criteria have a specific higher risk of developing psychotic disorders, whilst they are not at increased risk of developing other non-psychotic disorders. Among ARMS subjects, those meeting the BLIPS criteria have a distinct clinical outcome.

Financial support

This study was supported in part by a 2014 NARSAD Young Investigator Award to Paolo Fusar-Poli.

Ethical approval

Oxfordshire REC C (Ref: 08/H0606/71+5) for collection and analysis of data from the BRC Case Register (CRIS).

Disclosure of interest

The authors declare that they have no competing interest.