Selective reporting of results in guidelines

Taylor and Perera ¹ argue persuasively that the 2014 National Institute for Health and Care Excellence (NICE) schizophrenia guideline ² promotes cognitive–behavioural therapy (CBT) and other psychosocial interventions beyond the evidence. Its conclusions with respect to CBT also seem open to another charge, that of selective reporting: the highlighting of favourable results while unfavourable ones are suppressed. ³

In its clinical evidence summary (p. 232), NICE states that ‘when compared with standard care, CBT was effective in reducing rehospitalisation rates up to 18 months following the end of treatment’. NICE actually examined rehospitalisation rates in three of the large series (more than 100) of meta-analyses they carried out (data available at www.nccmh.org.uk). One of these compared CBT with standard care at up to 18 months and found a significant effect (5 trials, 910 patients, relative risk (RR) 0.76, 95% CI 0.61–0.94). Another compared CBT with standard care at 2–4 years and failed to find a significant advantage (2 trials, 513 patients, RR 0.82, 95% CI 0.64–1.05). The third meta-analysis compared CBT with ‘other active treatments’ (which consisted in all but one case of putatively inactive control interventions such as befriending and supportive counselling) at up to 2 years; this was again non-significant (5 trials, 506 patients, RR 1.07, 95% CI 0.86–1.33). The findings of the two negative meta-analyses are not mentioned in the NICE guideline. Neither does NICE mention that CBT was not found to be effective against relapse when compared with either standard care (3 trials, 460 patients, RR 0.85, 95% CI 0.50–1.41) or other active treatments (4 trials, 416 patients, RR 1.05, 95% CI 0.85–1.30). This omission is difficult to understand given the obvious relationship between relapse and rehospitalisation.

NICE goes on to state that ‘CBT was shown to be effective in reducing symptom severity as measured by total scores on items, such as the PANSS and BPRS, both at end of treatment and at up to 12 months' follow-up’. This was the case in the comparison between CBT and standard care, where there was a significant effect for CBT at the end of treatment (13 trials, 1356 patients, standardised mean difference (SMD) −0.27, 95% CI −0.45 to −0.10), as well as in meta-analyses of 6- and 12-month follow-up data. However, the findings were non-significant in the comparisons between CBT and ‘other active treatments’ both at end of treatment (6 trials, 396 patients, SMD −0.13, 95% CI −0.32 to 0.07) and at all follow-up points. Once again, NICE conveys an impression of uniform evidence of effectiveness against symptoms, whereas the reality is that an entire subset of pre-planned meta-analyses gave negative results.

Selective reporting arises when authors fail to publish data altogether, or when they arbitrarily decide which analyses and results to report in a publication. The NICE 2014 recommendations for CBT seem to be an example of the latter practice being applied to the results of multiple meta-analyses.