Negative results in phase III trials of complex interventions: Cause for concern or just good science?

Summary

Not all interventions that show promise in exploratory trials will be supported in phase III studies. But the high failure rate in recent trials of complex mental health interventions is a concern. Proper consideration of trial processes and greater use of adaptive trial designs could ensure better use of available resources.

Negative symptoms have a major impact on the quality of life of people with schizophrenia and antipsychotic drugs do little to relieve them. Promising results from exploratory trials of arts therapies raised hopes that they could help people with these symptoms. The negative results of a high-quality phase III trial of one such treatment – body psychotherapy – published in this issue are therefore disappointing. ¹ In this study, 275 people were randomised to 20 sessions of body psychotherapy or 20 sessions of Pilates. Attendance was good, but at the end of treatment negative symptoms were similar in both groups. These results echo those of a phase III trial of group art therapy that also failed to find clear evidence of patient benefit. ²

Arts therapies are not the only promising interventions that have failed to show benefit in phase III trials in mental health in recent years. Negative results have been reported in a succession of trials of complex interventions including training for parents of children with autism, web-based cognitive–behavioural therapy for depression, individual cognitive stimulation therapy for dementia and joint crisis cards and community treatment orders for people with psychosis. Although the principle of equipoise that provides the ethical basis for trials inevitably means that not all those tested will be found to be effective, negative results of so many trials of complex interventions in mental health requires critical consideration. Are negative results more common than in trials of pharmacological and other ‘simple’ interventions? What factors might explain null findings in phase III trials when exploratory studies showed benefit, and what can be done to ensure that as much value as possible is gained from these costly studies?

Reasons for differences

As summarised in Table 1, negative results in phase III trials of interventions that showed benefit in exploratory trials may reflect methodological weakness in either or both trials, or reflect genuine differences in the impact of the interventions when delivered at scale. The validity of trials may be compromised in multiple ways. Because it can be difficult to obtain funding for exploratory studies they are often run with limited budgets by committed ‘champions’ who were involved in developing them. Resource constraints may make it hard to minimise bias through steps such as masking researchers, minimising loss to follow-up and using independent teams to analyse study data. Small sample sizes in exploratory trials increase the chances of differences in baseline characteristics of patients, which increase the probability of confounding. In phase III trials, the measures needed to promote real-world validity may dilute effect: intervention fidelity may be compromised by implementation in diverse contexts by non-experts, practising in accordance with usual care. Thus, it is not surprising that even when interventions do show benefit in phase III trials, effect sizes are generally lower than those found in exploratory studies. ³ Small effect sizes found in exploratory studies may disappear altogether in the ‘noise’ of larger studies. ⁴

Table 1 Reasons for different findings in exploratory and phase III trials

Discrepancy	Possible reason
Differences in study design	Including, but not restricted to, differences in:
	outcome measures
	control treatment
	primary end point
False ‘positive’ result in the exploratory trial	Positive result produced by chance
	Positive result reflects bias
	Positive result reflects imbalance between study arms (confounding)
	Positive findings resulted from powerful placebo effects resulting from enthusiasm and sometimes pioneering efforts of researchers and therapists
False ‘negative’ result in the phase III trial	Negative result generated by chance (phase III study was underpowered)
	Negative result arises from failure to deliver treatment as intended (in terms of treatment content or context)
	Phase III trial recruited a population less likely to benefit from the intervention
	Negative result relates to measurement error (imprecise data on patient outcomes)
Intervention has efficacy but is not clinically effective	Typical therapists cannot deliver the quality of treatment over time that was provided by specialist centres during the first phase of developing the intervention
	Typical patients are unwilling to adhere to treatments to the extent that people do during short-term trials in specialist centres
	Treatment context (administrative support, out-patient care, transportation etc.) needed to ensure success of the intervention is not available in the wider healthcare system

Participant selection may be influential. People recruited to exploratory and phase III trials may differ in ways that shape engagement and effect. In exploratory studies participants are often recruited opportunistically from sites where investigators work. They may be more inclined to help the investigator/clinician or to trust the intervention being offered. The way that clinicians recruit patients for exploratory and phase III trials may also differ. For example, in a recent trial of community treatment orders, some clinicians outside academic centres were unwilling to take the chance that their most high-risk patients would be randomised to the control group, thereby excluding patients who may have benefited most from the intervention. ⁵

Measurement error, related to inconsistent application of research measures, may also dilute or mask the benefits of effective treatments. Phase III trials employ larger numbers of researchers across multiple sites, and it is therefore harder to assure that outcome measures are properly applied. Another potential cause of negative results is the precipitous conduct of phase III studies. Enthusiasm about an intervention together with pressures on academics to generate research income may lead to initiation of studies before a full understanding of active ingredients, mechanisms of action and the characteristics of patients who may benefit has been developed. The importance of such factors, particularly where complex interventions are concerned, is increasingly recognised. Rather than ask ‘what works?’, researchers are being urged to ask, what works for whom in which circumstances. ⁶ When exploratory studies show benefit, interventions should be tested for clinical effectiveness, but costly phase III trials should only be embarked upon once a theory of how the intervention generates benefits for patients has been developed. ⁷

Specific to complex interventions in mental health?

Although interventions in other areas of health can also be ‘complex’, psychosocial interventions in mental health are often relational in nature and rely even more on a clinician's ability to navigate interpersonal relationships and actively engage people in treatment. Do these factors have an impact on the likelihood of finding positive effects in phase III trials in mental health? To explore this we examined the final reports of phase III trials published by National Institute for Health Research (NIHR): Health Technology Assessment (the main funder of phase III clinical trials in the UK) between 2008 and 2014. During this period 71% (35 of 49) of trials of complex interventions returned negative results compared with 58% (21 of 36) for trials of drugs and other ‘simple’ interventions. Moreover, negative results were returned in a greater proportion of phase III trials of complex interventions in mental health (80%, 12 of 15) than other medical fields (68%, 23 of 34). Although not a statistically significant difference, the failure of so many complex interventions in mental health trials is striking, especially given the promising results of the exploratory trials that often preceded them.

Maximising returns from investment in trials of complex interventions

Process evaluations can add value to trials whether results are positive or negative. ⁸ In exploratory trials they can be used to develop understanding of recruitment, uptake of interventions and outcomes that may otherwise not have been considered, supporting design of larger-scale studies. ⁶ In phase III trials, well-designed process evaluations can be used to explore generation of outcomes, support interpretation and generalisation, and planning for further research. Qualitative data collected from researchers, referrers and study participants during the course of a phase III trial can be especially valuable. ⁹ For instance, Thornicroft and colleagues drew on accounts of clinicians and participants to explain the failure of joint crisis cards to reduce compulsory treatment. ¹⁰ With these data showing that contrary to the aim of the intervention, sessions to develop crisis cards were often led by clinicians, they concluded that if crisis cards were to work, they needed to be delivered by services that were willing to actively incorporate patient preferences in care planning.

Although historically tight word limits in paper-based journals have precluded detailed reporting of process evaluations with clinical trials, circumstances are changing. The advent of web-based reporting should make it easier for authors to disseminate and stakeholders to access findings of process evaluations alongside the main trial results, improving interpretation of the results.

Efforts to improve the cost-effectiveness of drug trials have led to the development of ‘adaptive’ trial designs. ¹¹ Such trials allow for pre-specified interim analysis and subsequent changes in design to stop early for success or futility, adjust drug doses or modify inclusion criteria. Such trials are not without their problems, not least the increased potential to generate false positive findings. ¹² However, at a time when so many trials of complex interventions are generating negative results we believe there is a case for greater use of adaptive designs in phase III trials of complex interventions. Adaptive trials could reduce unproductive investment in interventions that do not work and increase the likelihood that interventions that do work are tested among those who could benefit most from them.

Meanwhile, when considering negative results in phase III trials of interventions that appeared to demonstrate efficacy it is important to ask whether this is good science (a failure of an efficacious intervention to demonstrate clinical effectiveness), or whether methodological weaknesses in the exploratory trial or insufficient understanding of treatment process and context meant that it was premature to conduct a phase III trial.

Given the promise of exploratory studies and the pressing need to improve outcomes for people with schizophrenia, results of the trial of group body psychotherapy are disappointing. However, the team wisely conducted a parallel process evaluation. It will be interesting to see how these additional data can inform the development of new approaches to help people with psychosis and the design of future studies aimed at treating negative symptoms of schizophrenia.