Correspondence

Letter to the Editor

Routine psychological treatment and the Dodo verdict: a rejoinder to Clark et al. (2007)

The Dodo's verdict that ‘Everybody has won, and all must have prizes’ (Carroll, 1865/1946, p. 28; italics in original) was quoted more than 70 years ago by Saul Rosenzweig (1936) to describe the apparently equivalent outcomes of theoretically and technically diverse psychotherapies and has been cited and debated ever since. Based on our recent comparison of the outcomes of cognitive–behavioural therapy (CBT), person-centred therapy (PCT) and psychodynamic therapy (PDT) delivered in routine primary-care mental health practice within the NHS (Stiles et al. 2007), which replicated the results of an earlier study (Stiles et al. 2006), we drew the following conclusions:

For patients who completed the pre- and post-treatment forms, these therapies appeared effective, with mean pre/post improvements that approached those observed in efficacy trials. … The mean differences among the three targeted approaches, CBT, PCT, and PDT, did not approach significance, despite the high statistical power of this test. … The present results uphold the 70-year-old Dodo verdict. (Stiles et al. 2007, pp. 5–6)

A commentary by Clark et al. (2007) described our study as possibly including imbalances fatal to the outcome comparison and as drawing conclusions that were not warranted. Clark et al. (2007) offered three lines of argument. The main one suggested in several ways that NHS patients were not helped by these treatments, or at least were not shown to have been helped, so the question of whether one treatment helped more than another is moot. If nobody has won, then none deserve prizes. A second line of argument was that, as patients were not randomly assigned to treatments, some confounding factor other than the treatments might account for the results. A third line of argument was less specific, and it overlapped with the first two, but it seemed to be that naturalistic studies such as ours are generally inferior to randomized trials, so our results should be discounted.

Line of argument 1: Nobody has won

To support the position that the patients may have received no benefit from routine NHS treatment, Clark et al. (2007) argued (a) that our analyses included only a minority of the patients seen in these practices, (b) that the patients might have recovered without treatment, (c) that any improvement may have been attributable to medication, and (d) that the therapists may not have been properly trained or may have used inappropriate techniques, so the treatments may have been ineffective.

Argument 1(a)

Clark et al. (2007) argued, in effect, that because we restricted our analysis to patients who completed treatment, our sample was unrepresentative, and therefore our conclusions regarding effectiveness could not be generalized to all patients referred for counselling and psychotherapy in the NHS. Specifically, we did not include (i) patients who were in the therapists' caseloads but not included the database, or (ii) patients in the database who were not treated, or (iii) treated patients who failed to complete both pre- and post-treatment forms. We consider this argument as not germane. Clark et al. seemed to suggest that we had claimed to have assessed overall effectiveness of the NHS mental health system or that we were aiming to ‘provide the public with information about how likely they are to recover in a particular service’ (p. 4). They did this by framing their commentary with discussions of public disclosure of overall survival rates in cardiology units and analogous (but hypothetical) recovery rates from mental illness in psychotherapy services, by restating our conclusion without its qualifiers, as ‘all three treatments are effective’ (p. 2), and by describing the average for all patients – treated and untreated – as the ‘true recovery rate’ (p. 2). On the contrary, our analyses and our conclusions were explicitly restricted to patients who completed treatments of the designated types. We did not draw conclusions about treatment effectiveness for patients who remained untreated, or who dropped out of treatment, or who were treated but not assessed.

Among the patients who completed the targeted treatments we studied, the mean level of initial symptom intensity and the mean magnitude of improvement were of the same order as among patients who completed treatments in randomized trials to which Clark et al. (2007) referred in their commentary. To illustrate, we draw on published tables (Leach et al. 2006) for transforming scores on the Beck Depression Inventory (BDI; Beck et al. 1961) to equivalent scores on Clinical Outcomes in Routine Evaluation – Outcome Measure (CORE-OM; Barkham et al. 2001). Over the course of the treatments we studied (Stiles et al. 2007), the average CORE-OM score dropped from 17.6 to 8.8 (n=5613), as we reported, and from 19.0 to 9.4 among the patients (n=4984) who began above the clinical cut-off score of 10 – the inclusion criterion we used for assessing reliable and clinically significant improvement (RCSI). For comparison, CORE-OM-equivalent scores dropped from about 14.5 to about 8.7 among the patients who completed CBT treatment in the Deale et al. (1997) randomized trial of treatments for chronic fatigue syndrome and from about 17.1 to about 7.9 among the patients who completed CBT treatment in the Ehlers et al. (2003) randomized trial of treatments for post-traumatic stress disorder in patients who had failed to respond to a minimal intervention. These CORE-OM-equivalent scores were based on BDI changes from 14.5 to 8.9 and from 18.8 to 7.6 respectively and interpolated using the Leach et al. (2006) transformation tables. In contrast to our practice-based sample, patients were specifically recruited, assessed, diagnosed, and selected for these studies, and informed consent was obtained before they were randomly assigned to specified treatments; most or all of those assigned to CBT completed treatment. Others too have reported little difference between outcomes in practice-based studies and randomized trials (Benson & Hartz, 2000; Concato et al. 2000).

The suggested intention-to-treat logic, where the last data-point is carried forward to estimate post-treatment clinical status in individuals for whom post-treatment data are missing, has a different relevance in practice-based studies than in randomized trials, where the intention is counted only after patients have been assessed, selected, and assigned to groups. Of course, non-completion of treatment is a serious problem for NHS practice as well as for research, as we emphasized in our Abstract and our Discussion. We suspect, however, that non-completion in routine practice is more often attributable to personal, institutional, social, and economic conditions than to the theoretical approach the therapist uses. Our observation that those who complete treatment appear to benefit from it highlights the importance of overcoming the barriers to completing treatment.

As a check on differential dropout rates, we compared our six groups with patients who met our criteria for entry into those groups except that they did not complete post-session CORE-OM forms (see Table 1). Patients who met these criteria for the PDT+1 group (the ‘+1’ indicates that therapists employed one other therapy in addition to the targeted one) were somewhat more likely than average to complete post-treatment forms, whereas patients who met these criteria for the PCT+1 group were somewhat less likely to do so; the other four groups had completion rates within 2.5% of the average (χ²=52.47, df=5, p<0.001).

Table 1. Frequency and percentage of patients who did or did not complete a post-treatment CORE-OM and met other inclusion criteria (age 16, patient-completed pre-treatment CORE-OM, and therapist-completed End-of-treatment form indicating that patient met criteria for one of the designated groups as described by Stiles et al. 2007)

CORE-OM, Clinical Outcomes in Routine Evaluation – Outcome Measure; CBT, cognitive, behavioural, or cognitive/behavioural therapy; PCT, person-centred therapy; PDT, psychodynamic/psychoanalytic therapy; CBT+1, CBT combined with one other therapy; PCT+1, PCT combined with one other therapy; PDT+1, PDT combined with one other therapy.

Argument 1(b)

Regarding the possibility that patients would have recovered without treatment, we noted in our Discussion that CORE-OM and BDI scores remained reasonably stable for periods of up to a year among similar NHS patients waiting for therapy (Barkham et al. 2007). Contrary to Clark et al.'s (2007) speculation, the patients in our database were not mainly recent onset cases. They had waited an average of more than 60 days between referral and assessment. NHS psychotherapy seems not so easily accessible that patients can use it for transient problems. Among 3632 patients in our sample of 5613 who were rated as having depression, a problem sometimes cited as changing quickly, 82% were rated as having had the problem for 6 months or more.

Argument 1(c)

To address the question of whether the observed improvement was attributable to medication, we did comparisons based on the therapist's response to the item ‘Is the client currently prescribed medication to help with their psychological problem(s)?’ on the initial assessment form (completed after the first session). A ‘yes’ response thus included patients whose prescriptions were given near the beginning of therapy as well as those who had taken the medication for longer. Therapists provided this information for 5442 (97%) of the patients we studied, responding ‘yes’ for 2989 (54.9%) and ‘no’ for 2453 (45.1%). These prescriptions reflected professional judgements regarding each case, not random assignment to drug treatment condition.

On average, patients who had such prescriptions began treatment with higher CORE-OM scores [18.4 v. 16.6, t(5440)=10.74, p<0.001] and ended treatment with higher CORE-OM scores [9.3 v. 8.1, t(5440)=6.96, p<0.001], having changed by a larger amount [9.1 v. 8.5, t(5440)=3.41, p<0.001] than patients who did not have such prescriptions. However, patients who were not prescribed medication, had a higher rate of achieving RCSI than patients who did have such prescriptions (62.6% v. 55.1%, χ²=27.07, p<0.001, n=4806) (for this analysis, we included only patients who began above the clinical cut-off). This seeming paradox probably shows less about effects of medication than about more general properties of change scores: patients who begin with relatively high scores tend to change by a larger amount but nevertheless fail to reach as low a final score (and so are less likely to fall below the clinical cut-off, to end in the normal range, as required to achieve RCSI) as patients who begin with relatively low scores (e.g. Minami et al. 2007). In any case, the differences in the mean change scores and the RCSI rates were modest. Patients tended to improve substantially whether or not they were prescribed medication.

We also examined the distribution of patients prescribed or not prescribed medication across our six target treatment groups (see Table 2). The distribution was not random (χ²=23.21, p<0.001, df=5); the PDT and particularly the PDT+1 groups had somewhat smaller proportions of patients prescribed medication in comparison with the other groups. We return to this observation shortly.

Table 2. Frequency and percentage of patients taking prescribed medication at first session

CBT, Cognitive, behavioural, or cognitive/behavioural therapy; PCT, person-centred therapy; PDT, psychodynamic/psychoanalytic therapy; CBT+1, CBT combined with one other therapy; PCT+1, PCT combined with one other therapy; PDT+1, PDT combined with one other therapy.

Argument 1(d)

We did not, as Clark et al. (2007) suggested, ‘acknowledge that the quality of the treatments delivered may have been poor’ (p. 4). The magnitude of the pre-post improvements suggests the quality was adequate. We said we had no independent evidence of how the treatments were delivered, though as we noted, for better or worse, they represent the way these approaches are currently being practised (Stiles et al. 2007).

The commentary's authors are publicly known proponents of CBT, and they seemed to suggest that deviations are more likely or more serious for CBT than for the other approaches. As we noted in our Discussion, ‘proponents may be differentially sensitive to possible deviations within their favoured approach’ (Stiles et al. 2007, pp. 13–14). Our experience and colleagues' reports indicate that those who identify with PCT and PDT are at least as concerned about inexpert execution of their approaches in routine practice as are those who identify with CBT. The assertion that ‘PCT is fairly easy to deliver within the constraints of primary care but CBT is not’ (Clark et al. 2007, p. 4) is likely to be as objectionable to PCT proponents as the suggestion that ‘the “CBT” involved teaching the patient “coping techniques” such as going into a toilet cubicle before a public speaking event and pushing his arms hard against the walls to get “psyched up” ’ was to one of the commentary's authors (Clark et al. 2007, p. 3).

Line of argument 2: Possible confounds

According to the logic of randomized trials, if groups are not equivalent initially, then any differences observed following treatment might be attributable to the initial non-equivalence. Insofar as we observed no outcome differences, an alternative to the Dodo verdict would require a scenario in which some initial non-equivalence differentially penalized one approach that would otherwise have proved superior.

In psychotherapy research, the best predictor by far of post-treatment scores is usually pre-treatment scores on the same measure. Although patients were not randomly assigned to treatments in our studies, the groups had equivalent initial scores on the CORE-OM, so any hypothetical scenario must be built on some other initial difference.

We addressed some of the most likely confounds in our paper (see tables 3 and 4 of Stiles et al. 2007). Two others are addressed in Tables 1 and 2 of this rejoinder. Based on the latter results, one might argue that the PDT+1 treatment had a latent superiority insofar as its mean pre-post improvements matched the other groups even though a slightly larger proportion of its patients completed treatment (Table 1) or because it faced a slightly larger proportion of unmedicated patients (Table 2). On the other hand, one might argue that the PDT+1 treatment had a latent inferiority insofar as unmedicated patients tended to have higher RCSI rates.

In their commentary, Clark et al. (2007) raised the further possibility of differences in chronicity, and they focused on the observation that PCT and PDT were more likely to have interpersonal problems and bereavement/loss, according to their therapists. Our data on problems and chronicity are too complex to analyse for this rejoinder (e.g. separate ratings for each of multiple problems), but we know of no serious differential distributions. There is evidence that interpersonal problems respond to treatment more slowly than does symptomatic distress such as depression (Barkham et al. 2002), so one might argue that PCT and PDT had some latent superiority in achieving equivalent outcomes despite confronting more of such problems. On the other hand, the differences in interpersonal problems could reflect differential reporting by therapists using different approaches rather than differential selection of patients.

Although we agree in principle that some confounding variable could differentially handicap one or another of the approaches, masking its superiority, the differences we observed were very modest, and we have not found any scenario presented to be compelling.

Line of argument 3: Superiority of randomized trials

Clark et al. (2007) devoted a good deal of space to reviewing randomized trials in which cognitive therapies were found superior to various comparison conditions. Other parts of the commentary offered procedural suggestions, such as regular monitoring of progress in routine practice. The implication of these comments seemed to be that because there are virtues in research designs and procedures that our study did not use, our study's results should be discounted. We agree that there are virtues in these alternatives, but we don't see how their existence impugns our conclusions. We further believe our practice-based design has virtues of its own, particularly including its sample size and its realism (external validity) in ‘address[ing] the effects of treatments as routinely delivered, using practitioners’ versions of the treatments and the patients who typically receive them' (Stiles et al. 2007, p. 2).

The commentary's closing story about early misinterpretation of research on hormone replacement therapy was an instructive cautionary tale about how self-selection may distort results in non-randomized groups, but we were unsure how it was meant to apply in this situation beyond the earlier point that unappreciated confounds are possible.

In response to a possible broader implication that randomized trials represent a gold standard that trumps all other methods, we note that randomized trials of psychotherapy are also a subject of debate (e.g. Persons & Silberschatz, 1998; Wessely, 2007) and have their own difficulties of execution and interpretation (Haaga & Stiles, 2000). For example, a review by Luborsky et al. (1999) concluded that the investigators' allegiance to a favoured treatment could account for over two-thirds of the outcome variance in comparative studies. Randomized trials were designed for drug comparisons, where double-blind procedures can balance placebo and allegiance effects, while individual patient differences are addressed by randomization and patient participation is limited to adherence to the prescribed pill-taking regimen. In psychotherapy research, by contrast, double-blind designs are not possible, and the patient cannot plausibly be considered as passive recipient of a standard dose administered via a neutral delivery vehicle. Psychotherapy treatment manuals (even cognitive therapy manuals) emphasize clinical judgement, responsive application of treatment principles, and active, extensive patient collaboration and participation. Psychological changes (improvement or deterioration) can recursively feed back to influence the treatment procedures, so that the execution of the treatment becomes conditional on its emerging outcome, a process that is problematic (some would say fatal) for interpreting an independent variable's effects (Stiles et al. 1998).

A few corrections

We note here a few errors in Clark et al.'s (2007) summary of our method, hoping to prevent their further propagation. Clark et al. (2007) said the therapists anonymously submitted their questionnaires to a central database (p. 1), whereas the questionnaires were collected for clinical and administrative purposes, not for research, and only subsequently anonymized for the database (see Mellor-Clark & Barkham, 2006). Clark et al. (2007), said, ‘the median number of cases submitted [to the database] by each therapist was only six’ (p. 1), whereas six was the median contributed to the final sample. In the full data set (n=33 587 patients), 637 therapists contributed a median of 17 cases. Many therapists contributed on only one or a few cases, however, and, as we indicated, the 343 therapists who saw 15 or more of the patients contributed 31 966 (95%) of the patients in the full dataset. Clark et al. (2007) said the ‘5613 submitted constitutes 38% of the patients who were submitted to the database’ (p. 1), whereas we explained that the ‘percentage of patients [who] returned valid pre-treatment and post-treatment CORE-OM forms [was] 38% or 12 746/33 587’ (Stiles et al. 2007; p. 7). Finally, the items about presenting problems were not the end-of-treatment form as Clark et al. (2007) said (p. 3), but on the initial assessment form.

Conclusions

1. (1) In both studies (Stiles et al. 2006, 2007), the six groups' mean outcomes were remarkably similar to each other, fuelling our speculation that outcomes may be regulated by responsive, self-organizing processes (see our Discussion as well as Stiles et al. 1998) rather than determined by treatment assignment, as assumed in randomized trials. Taking this seriously would represent a radical shift in thinking about assessing outcomes of alternative treatments.

2. (2) We strongly agree that caution is warranted in interpreting any observations and that all scientific findings should be regarded as tentative. Cautiously and tentatively, however, we suggest that if these routine treatments were ineffective for those who completed them, we would not have seen the evidence of improvement that we observed. And if there were large real differences in the effectiveness of the treatments we compared, we would have seen evidence of those differences in our analyses.