Network meta-analysis: a useful tool for comparing different psychological treatments for panic disorder?

Summary

Network meta-analysis (NMA) is a statistical method that allows comparison of multiple treatments against each other. The use of NMA, and its advantages and potential limitations, is discussed here in the context of this issue's Cochrane review of psychological treatments for panic disorder. The review showed benefits for psychological treatment when compared to a waiting list, particularly for cognitive behavioural therapy in both the short and longer term. The findings were limited by the primary studies available, but despite these, this analysis using NMA presents the best evidence to date of the relative benefits of different types of psychological treatments in panic disorder.

DECLARATION OF INTEREST

None.

Panic disorder (defined by the presence of recurrent unexpected panic attacks: American Psychiatric Association 2013) is common in the general population, where lifetime prevalence is 1–4% (Eaton 1994), rising to 10% in primary care (King 2008). Agoraphobia (characterised by anxiety about actual or anticipated exposure to places or situations from which escape might be difficult or embarrassing: American Psychiatric Association 2013) is commonly associated with panic disorder. In the general population about one-quarter of people suffering from panic disorder also have agoraphobia, and this proportion is much higher in clinical samples (Kessler 2006). The presence of agoraphobia in panic disorder is associated with increased severity and worse outcomes (Kessler 2006). Panic disorder is more common among women than among men, with a 2:1 ratio, rising to 3:1 in panic disorder with agoraphobia. It is highly comorbid with other psychiatric disorders, such as drug dependence, major depression, bipolar disorder, social phobia, specific phobia and generalised anxiety disorder (Grant 2006).

Guidelines published by the National Institute for Health and Care Excellence (NICE) recommend psychological therapy, antidepressant medication and self-help (National Collaborating Centre for Mental Health 2011), with psychological therapy having evidence for the longest duration of effect. Although the NICE guidelines recommend cognitive–behavioural therapy (CBT), other psychological therapies have been proposed, including: psychoeducation, supportive psychotherapy, physiological therapies (e.g. breathing retraining, relaxation techniques and biofeedback), behavioural therapy, cognitive therapy, third-wave therapies (e.g. mindfulness-based cognitive therapy, acceptance and commitment therapy, compassionate mind training, extended behavioural activation, metacognitive therapy and schema therapy) and psychodynamic therapies. The NICE guidelines suggest that CBT should be delivered in 1- to 2-h weekly sessions, for a total of 7–14 h, within 4 months. However, the effect of variables such as the number of sessions, duration of treatment and the therapeutic setting (group v. individual, face-to-face v. remote) are still unclear.

Previous evidence

A previous Cochrane meta-analysis (Furukawa 2007) compared psychological therapy combined with antidepressants v. either treatment alone and showed the superiority of combined therapy in the short term (v. antidepressant pharmacotherapy: risk ratio (RR) 1.24, 95% confidence interval (CI) 1.02–1.52; v. psychotherapy: RR 1.17, 95% CI 1.05–1.31) and of combined therapy and psychological therapy alone in the long term (combined therapy v. pharmacotherapy alone: RR 1.61, 95% CI 1.23–2.11; combined therapy v. psychotherapy: RR 0.96, 95% CI 0.79–1.16). Behavioural therapy and CBT showed the strongest evidence. Another meta-analysis analysed the efficacy of different psychological interventions (Sánchez-Meca 2010), and showed results in favour of the combination of exposure strategies with relaxation training, or breathing retraining techniques, or both (exposure plus relaxation and breathing v. placebo: mean effect size 1.837, 95% CI 1.265–2.409). However, the analysis revealed substantial heterogeneity among included studies, particularly in the type of therapy and type of control group.

Thus, the evidence supports the efficacy of psychological treatments in general for panic disorder, but does not reveal which specific psychological treatment is the best, what aspects of the treatments are the most efficacious, and how long their benefits might last. A standard meta-analysis (as seen in the previous reviews) uses separate pair-wise comparisons of treatments. So, when reviewing the efficacy data of the many possible psychological treatments for panic disorder, the clinician is presented with an array of separate pair-wise comparisons. In addition, some comparisons between treatments have not been studied directly and so the head-to-head comparison cannot be assessed. Network meta-analysis (NMA, also called multiple treatments meta-analysis) is a method for comparing a number of different treatments in a single analysis (Mavridis 2015), and can address some of these problems. It is a statistical method that allows multiple treatments to be assessed at the same time and uses direct and indirect evidence from the comparison data available (Leucht 2016). Box 1 gives further explanation.

BOX 1 Network meta-analysis (NMA)

• • A statistical method based on the simultaneous analysis of direct evidence (comparing treatments within the same study) and indirect evidence (comparing interventions across different studies using a treatment in common: see Fig. 1 for an explanation).

• • Indirect evidence is important. It provides data on comparisons for which no trials exist and, in combination with direct data, it also improves the precision of the estimates, producing so-called ‘mixed evidence’ (Cipriani 2013).

• • Data from all the available trials can be presented as a network (Fig. 2). The method also allows a calculation of a probabilistic ranking (somewhat similar to a mileage table in a road atlas) to help the identification of interventions that are more effective than others on different outcomes.

• • Examples of NMA reviews in psychiatry include antimanic drugs in acute mania (Cipriani 2011), maintenance treatments for bipolar disorder (Miura 2014), antidepressants in acute treatment of unipolar major depressive disorder (Cipriani 2009; Furukawa 2016) and antipsychotic drugs in acute treatment of schizophrenia (Leucht 2013). NMAs have increasingly been employed to support clinical guidelines and health technology appraisals (Barbui 2011).

The current review

Methods

In the review in this month's Cochrane Corner, Pompoli et al (2016) used NMA to compare eight different forms of psychological therapy and three forms of a control condition. This review, along with several others, contributes to a wider group of Cochrane reviews on panic disorder (Furukawa 2007; Watanabe 2009; Xiao 2011; Guaiana 2013a b ; Imai 2014; Bighelli 2016).

The review authors aimed to answer two main questions: (a) whether any of the psychological therapies were more effective or better tolerated than others in the rapid reduction of panic/agoraphobia symptoms, and (b) whether any of the psychological therapies show better results 1 year later. They searched (until March 2015) for all randomised controlled trials (RCTs) involving out-patients over the age of 18 with a primary diagnosis of panic disorder with or without agoraphobia according to standard diagnostic criteria. Cross-over RCTs were included (but only the first randomisation period, to avoid possible carry-over effects in the second phase). They included both previously treated and treatment-naive patients. Comorbidity with other anxiety disorders was allowed, if separate results for panic disorder were reported.

The reviewers evaluated all the eight therapies listed with three comparator interventions. The choice of comparator condition in trials of psychological treatment is particularly difficult (Nutt 2008), and the authors categorised the comparator as ‘no treatment’, ‘waiting list’ or ‘attention or psychological placebo’ (APP) (Box 2). Therapies could be of any length, individual or group, but had to be delivered face to face. Combinations of psychological therapies were excluded. Pharmacological co-administration was allowed provided that there were no systematic differences between the study arms.

BOX 2 Comparator interventions

• • Comparator interventions (acting as a ‘control’ group) are a key issue in trials of psychological treatments, as there are different interpretations of what care the comparator group should receive to reduce confounding variables.

• • In this review, the trials used the following comparators: no psychological treatment (participants received assessment only and they knew that they would not receive the active treatment in question after the trial), waiting list (assessment, and they knew that they would receive the active treatment after the waiting phase), attention or psychological placebo (APP: participants received a face-to-face inactive intervention, e.g. meeting for non-directional conversation). No-treatment and waiting-list comparators are both problematic (Goodwin 2016). Any benefit for active treatment may be due to (or amplified by) a potential ‘nocebo’ effect (undesirable effects that are or could be caused by an inactive treatment) for the no-treatment and waiting-list conditions. It can be argued that a claim for efficacy of a therapy requires comparison with a control procedure that engages patients and therapist in a relationship which omits only the active ingredients of the therapy. APP may meet this criterion, although the ‘inactive intervention’ needs to be chosen carefully to avoid active elements, but maintain the potential positive effects of meeting a therapist regularly.

• • Definitions of comparator interventions among different studies in this review were generally inconsistent, and the description given by authors was often not detailed enough to allocate to one of the three groups. Particular inconsistency existed in the definition of ‘treatment as usual’ (TAU) which was used in some studies and was poorly specified, including variously no treatment, waiting list or supportive psychotherapy.

Primary outcomes were short-term (within 6 months) remission or response of panic disorder (according to standardised rating scales such as the Clinical Global Impression (CGI) scale or the Panic Disorder Severity Scale (PDSS)), and drop-out (for any reason) as a proxy for treatment acceptability. Secondary outcomes were short-term improvement of panic disorder and long-term remission or response of panic disorder. The reviewers used an intention-to-treat analysis, assuming that all randomised patients for which outcome data were not available were non-responders.

Results were presented as summary-of-findings tables for each of the main pair-wise comparisons (as would be done in a standard meta-analysis reporting each pair-wise comparison). In addition, results were also presented as a summary table of the overall results of the NMA by presenting the ranking of treatments yielded by these analyses for each outcome as represented by their SUCRA (surface under the cumulative ranking curve) value (Salanti 2011), derived from the NMA. The tables included an assessment of quality according to the GRADE criteria adapted for use in an NMA (Salanti 2014). The reviewers also carefully addressed methodological issues that need to be considered in an NMA analysis (Box 3).

BOX 3 Methodological issues in network meta-analysis (NMA) and their relevance in this review (Pompoli 2016)

NMAs are becoming increasingly popular but there are methodological issues to consider.

• • Randomised evidence is used for both direct and indirect comparisons. Indirect evidence preserves the original randomisation, but the indirect evidence is not itself randomised, as treatments have originally been compared within but not across studies. Direct evidence is therefore more robust and indirect evidence should ideally be used as a supplement to direct evidence.

• • The shape of the network is also critical. Ideally, it should be well connected, but this is not always so. In this review, there appeared to be evidence in favour of supportive psychotherapy but this was directly explored only in the comparison with CBT. The comparison of supportive psychotherapy v. all other treatments was informed only by indirect evidence.

• • A central assumption of a well-conducted NMA is that the subject data meet the criterion of ‘transitivity’: the concept that any patient within the network could have been randomised to any of the treatments. In other words, the criteria for selection of patients for the different studies need to be similar and the degree to which this holds should be addressed by the reviewers. In this review, the reviewers compiled a table of important trial and patient characteristics and inspected the similarity of factors considered likely to modify treatment effect. They also assessed the inclusion and exclusion criteria of every trial in the network to ensure that patients, trial protocols, etc. were similar in aspects that might modify the treatment effect.

• • NMA allows consistency to be examined statistically: if A is better than B and B is better than C, is A better than C? If not, there is inconsistency in the loop between A, B and C. In this review, inconsistency was examined for all individual comparisons and across the global network. The reviewers also described a clear strategy for dealing with inconsistency and effect modifiers where these were found.

• • As with all syntheses, the quality of the evidence is crucial to interpretation of the results of an NMA. The reviewers assessed quality using methods from the GRADE working group for all primary outcome estimates.

Results

In total, 60 RCTs were included in the qualitative analyses and 54 (n = 3021) of these in the quantitative analyses. In all but one of the studies the participants were out-patients. No studies were able to ensure blinding (masking) of treatment allocation for the therapists. Eight studies were considered to be at low risk of bias in the blinding of the outcome assessors, 22 studies to be at high risk and the remaining studies did not report enough information to make a judgement.

Among experimental interventions, CBT was by far the most studied (42/54 studies). In the majority of studies, the control condition was represented by a waiting list (30 studies); APP was used in only 3 studies and no treatment in only 2. Psychological therapy was administered individually in 18 studies and in a group in 13. The average number of sessions was 10 and they were weekly in almost every study. Therapists were specifically trained in the administered intervention in most of the studies (n = 37). Only 25 of the studies specified the percentage of patients receiving a drug therapy; 30 studies required patients not to change medication dosage during the study. There was great variability in the outcomes measured and instruments used. These included panic frequency, the Anxiety Sensitivity Index (ASI), Body Sensations Questionnaire (BSQ), Agoraphobic Cognitions Questionnaire (ACQ), Panic Disorder Severity Scale (PDSS), Mobility Inventory for Agoraphobia (MI), Fear Questionnaire (FQ), State–Trait Anxiety Inventory (STAI) and Beck Anxiety Inventory (BAI).

The quality of the evidence for the entire network was found to be low to very low for all outcomes, with the presence or likelihood of bias in many domains. The networks appeared to be well connected, but were generally underpowered to detect any important disagreement between direct and indirect evidence (see Fig. 1 for explanation of these terms).

FIG 1 The combination of direct and indirect evidence into a single effect size for treatment A v. treatment B (mixed estimate) (Smith 2016). An indirect comparison allows an estimate of the effect of treatment B relative to treatment A via a common comparator C by statistically combining the summary effects from ‘A v. C’ and ‘B v. C’ studies (Caldwell 2005). NMA combines direct and indirect evidence across a network of studies to make inferences regarding the relative effectiveness of multiple interventions.

FIG 2 An example of a network arising from a network meta-analysis (NMA). Each node represents an experimental or control condition; comparisons explored in the included trials are represented by lines connecting the nodes. The size of the node represents the number of participants involved in that condition, and the width of the line, the number of participants involved in that comparison. The network only shows the direct evidence; this is supplemented by indirect evidence that can be inferred by comparing evidence with a common comparator, such as a waiting list. (Adapted from Cipriani et al (2016), with kind permission of Professor A. Cipriani.)

Overall, the results showed the superiority of psychological therapies over the waiting-list condition. The NMAs for short-term remission, response and improvement all showed well-replicated evidence in favour of CBT, as well as some evidence in favour of psychodynamic therapies and supportive psychotherapy over other therapies. For example, the results favoured CBT over waiting list for short-term remission (18 studies, for pair-wise meta-analysis (direct evidence only): odds ratio OR 0.13, 95% CI 0.07–0.22; for NMA (incorporating direct and indirect evidence), OR 0.12, 95% CI 0.07–0.21; for NMA adjusted for small study effects (SSE), OR 0.33, 95% CI 0.16–0.69). In terms of short-term drop-out, psychodynamic therapies ranked higher than CBT (OR 0.52, 95% CI 0.15–1.8) and a high ranking was also achieved by third-wave CBT.

In the long term, CBT showed the highest ranking for treatment response, followed by psychodynamic therapies, suggesting that the effects of these two treatments may be more stable than other psychological interventions. The superiority of CBT in the long term may be due to the ‘relapse prevention’ component, but this could not be specifically addressed by the review. In addition, it was not possible to explore the effects of concomitant drug therapy on long-term remission/response.

Conclusions

To date, this is the most extensive and methodologically rigorous systematic review of psychological therapies for the treatment of panic disorder, and it is the first to use NMA to compare a number of these therapies for the disorder with each other. The results are in line with the findings of previous pair-wise meta-analyses (Mitte 2005; Sánchez-Meca 2010; Bandelow 2015; Imai 2016).

The ranking of treatments according to the SUCRA values derived from NMAs showed that CBT was often superior to other therapies, but the effect sizes were small. The two studies exploring psychodynamic therapies showed promising results, but further research is needed to compare CBT and psychodynamic therapies directly. Behavioural therapy does not appear to be a valid alternative to CBT as a first-line treatment for panic disorder with or without agoraphobia.

Unexpectedly, there was some evidence in support of non-specific supportive psychotherapy, but these results should be interpreted cautiously. The efficacy of supportive psychotherapy was explored directly only in the comparison with CBT, and the comparison of supportive psychotherapy with all other treatments was informed only by indirect evidence.

Methodological problems relating to the use of the NMA statistical techniques were well addressed by the reviewers (assessing the principles of transitivity and consistency: Box 3). However, despite a methodologically sound NMA, the reviewers could not exclude the general problems associated with syntheses of psychotherapy trials (Cipriani 2017). These include the small scale of the primary studies, allegiance bias (i.e. investigators may be heavily invested personally and professionally in showing that their treatment works), lack of blinding and publication bias (Flint 2015). In addition, the possible harms of both active and comparator treatments are not collected systematically (Nutt 2008). By considering only benefits, without balancing a true assessment of harm, the selection of treatments may be inaccurate.

Although the findings are limited by the quality of the primary studies, this NMA presents the best evidence to date of the benefit of psychological treatments in panic disorder. It also suggests that there is no further need to explore the comparison between CBT and waiting list, nor between CBT and behavioural therapy. However, future studies should explore active treatments directly, especially psychodynamic therapies and supportive psychotherapy v. CBT, with clearly delineated methodology, larger sample sizes and longer-term follow-up. In addition, it would be helpful to explore which components of CBT contribute specifically to its benefits.