Over-simplification and exclusion of non-conforming studies can demonstrate absence of effect: a lynching party?

A number of meta-analyses of cognitive behavioural therapy (CBT) in severe mental illness have been published by its advocates (Wykes et al. 2008), independent commentators (Jones et al. 2004) and now finally by its opponents (McKenna, 2001, 2006; Turkington & McKenna, 2003). Lynch et al. (2009) conclude that there is a small effect of CBT in severe depression but no effect in bipolar disorder and schizophrenia.

In bipolar disorder, there have been relatively few studies and the largest reported a negative finding which has overshadowed promising earlier findings. This may be related to the treatment group selected in that study or the therapeutic intervention used. Lam et al. (2003) have described an adaptation of CBT, whereas the intervention in the larger study seems to have been more behavioural in form, concentrating on early intervention, treatment adherence and relapse prevention (Scott et al. 2006).

The findings on schizophrenia are more interesting. Meta-analyses and reviews published so far have found favourably for CBT. Lynch et al. (2009) find differently. For relapse prevention, this may be because a large new study has produced negative findings in this area and also because of the use of studies that do not fit their inclusion criteria. For example, Hogarty's Personal Therapy has very different origins and practice from CBT and was not described as such by him (Hogarty et al. 1997). Most importantly, the total exclusion of studies using hospitalization as a proxy for relapse led to a substantial underestimate of effect.

Lynch et al. (2009) review studies which have an active control but wrongly conclude that such controls ‘lacked any specific therapeutic effects’. Befriending, for example, appears to have positive effects on delusions including paranoia but not hallucinations (Samarasekera et al. 2007). Active controls allow for differentiation of effects from the non-specific but very important effects of developing a relationship with patients. Studies comparing with treatment as usual are summarily dismissed but these do have relevance in assessing generalizability in effectiveness studies.

Studies in this area have used different time scales and target symptoms and previous meta-analyses have made allowance for this. Lynch et al. (2009) do not, focusing only on end-of-treatment scoring. This has allowed them to claim that ‘perhaps the best study published to date … conducted by Sensky and colleagues’ (Beck et al. 2008) was one which failed. It did not indeed show a difference on most measures at the treatment end-point – apart from suicidality (Bateman et al. 2007) – but such effects were apparent at the 9-month follow-up and maintained at 5 years (Turkington et al. 2008). Lynch et al. (2009) justify excluding these beneficial effects because some other studies did not show this despite others – not quoted – which did (Drury et al. 2000; Turkington et al. 2006).

It is also disingenuous to hold up double-blind placebo trials of medication as if they were infallible: the effects of bias and other factors are still clearly a major concern as has emerged with evaluation of the latest generation of antipsychotics (Tyrer & Kendall, 2009). Blindness is rarely assessed in these trials. Yet the side-effects of drugs such as haloperidol compared with olanzapine and quetiapine are markedly different such that it would be expected that patient and rater would frequently be aware of which drug was being provided. Generalizability is also a major concern – recruiting patients with delusional beliefs, especially paranoia, to any study is difficult but to medication trials especially so.

The approach of Lynch et al. (2009) to mental disorder using concepts and terms such as schizophrenia (van Os, 2009) is also being increasingly recognized as too blunt for psychosocial interventions. Successful targeted studies are emerging in early psychosis (Morrison et al. 2004) and where psychosis is associated with substance abuse (Haddock et al. 2003), command hallucinations (Trower et al. 2004), post-traumatic stress disorder (Mueser et al. 2008) and anger (Haddock et al. 2009). Such diversity makes meta-analysis that much more complex and none of these studies was included in this meta-analysis. Over-simplification and exclusion of non-conforming studies can readily demonstrate limited or absence of effect. However, the acceptability of CBT to patients, carers as well as practitioners suggests that positive findings in the clinical studies undertaken so far are valid and generalizable in clinical practice. These are now being used and evaluated further in countries from China to Pakistan to the USA.