An effective intervention delivered at sub-therapeutic dose becomes an ineffective intervention

Summary

Evidence to date indicates that the individual placement and support model of supported employment helps people with mental illness to obtain competitive jobs. The study by Howard et al (this issue) is the first unsuccessful trial of this model. Vocational workers had far fewer contacts with clients and employers than normal.

Consistent evidence in support of the IPS model of supported employment

There is widespread agreement that meaningful work is an important factor in the recovery of people with mental illness. The desire to promote social inclusion, while also providing opportunities for meaningful work, leads to the favouring of approaches that aim to help people with mental illness find work in the open market. Among such approaches the most effective, and the one recommended at the policy level in England, ¹ is the individual placement and support (IPS) model of supported employment. ² More than a dozen randomised trials of IPS, most conducted in the USA but also in Australia, Canada and Hong Kong, ³ as well as a six-centre European trial, ⁴ have all concluded that IPS programmes achieve better competitive employment outcomes for their clients than traditional programmes. ^4,5 A synthesis of the results of 11 of the trials reported an average of 61% of clients attaining competitive employment over the study follow-up period compared with 23% for controls. ⁵ In addition to these randomised trials, more than ten additional studies from the USA, Canada and England, relying on variety of other research designs, all reported significantly better competitive employment outcomes for IPS. ⁶

Discordant findings from the SWAN study

The Supported Work and Needs (SWAN) study by Howard et al published in this issue of the British Journal of Psychiatry is the first to report no significant benefit from programmes described as following the IPS model. ⁷ The authors report having attained good model fidelity ratings – 67/75 in one borough and 69/75 in another, both considered consistent with good implementation. Yet the percentage of IPS group clients who obtained competitive employment (13%) is the lowest ever reported, and there is no statistical difference between the IPS and the control groups. Among randomised trials the next lowest comparable percentage is 27%, and this in inner-city Baltimore, including clients who never expressed any interest in competitive employment. ⁸ In the six-centre European trial, employment rates in the IPS condition ranged from 34% in Groningen to 82% in Sofia; in London they reached 48%. ⁴

Good model fidelity scores… but employment workers who rarely saw employers or clients

The authors offer a number of possible explanations for their unexpected finding: that IPS may be less effective in non-US settings because of differences in benefit programmes or other contextual factors; that participants might have been less motivated to work and more impaired than in other trials; that their IPS programme was implemented by a voluntary sector provider that was not structurally or managerially integrated with the community mental health services; that the number of contacts made by IPS staff appeared low; and finally that employers might have been more reluctant to employ people with mental illness than in other studies. Taken together, these factors may well explain the striking contrast between the findings of this trial and those of others. Among these factors, however, the low number of contacts that IPS staff made must have had a determinant role.

The authors state that the mean number of contacts made by IPS workers on behalf of the 73 clients who were followed up and who received any IPS service over 1 year was 14, including contacts with employers and other third parties (9.3 if the entire IPS group is taken into account). This corresponds to hardly more than one contact per month. Reflecting the low intensity of the intervention, the authors report a cost of UK£296 per client per year, including a third of clients who did not actually access IPS services, which translates to about £442 per client who used IPS services and who was followed up. A US study, using 2001 data from seven sites that had implemented IPS with a high level of fidelity, concluded that IPS programmes cost about US$2449 per full-year-equivalent client on average, ⁹ which translates to about £2000 in 2009 after adjusting for inflation and converting currencies using purchasing power parities. Looking at it another way, four IPS workers served 73 clients – an average of 18 each. Following the authors in taking £40 000 as an annual cost per employment worker, the cost of IPS services should have exceeded £2200 per full-year-equivalent client – five times more than the estimated figure. The intensity of the intervention provided in the SWAN study thus appears to have been only about a fifth of what one would have expected, even among the two-thirds of clients who did engage with the programme, disregarding the third who did not. It is hardly surprising that an intervention of such low intensity should have had little or no effect.

Lessons to be learned

One lesson to be learned from this experience is methodological. The fact that both programmes were able to attain a reasonable score on the IPS fidelity scale reminds us of the limitations of such scales. Although the construct validity of the IPS fidelity scale is to some extent validated by the number of studies that indicate a relationship between fidelity and outcomes, some items play a more crucial part than others – yet they are weighted equally in the scale (even in its newer version). In the present case it may be that employment workers served many more clients than those in the study, which would at worst have reduced the total score by 4 points on a scale that has a maximum of 75 points. It is clear that increasing case-load indefinitely, or equivalently reducing the intensity of interventions to near zero – no matter how much in conformity to the model the intervention might be otherwise – cannot yield good outcomes. As the number of supported employment programmes increases, it will become more feasible to refine the scoring on fidelity scales so that they are more closely predictive of outcomes. Until this is done, programmes that can hardly be described as IPS will nonetheless be able to describe themselves as such on the basis of a reasonable fidelity score.

At a policy level, the SWAN study illustrates how challenging good programme implementation can be. There is now a wealth of evidence to show that, for example, naive ‘spray and pray’ methods of training staff – simply providing start-up training – are ineffective. ¹⁰ Much more elaborate mechanisms are needed, involving, for example, on-site coaching over a year or more by consultant trainers grouped within a technical assistance centre, and systematic monitoring of fidelity and outcomes. ^11,12 The Evidence-Based Practices Project has shown that such carefully designed mechanisms can enable a significant percentage of programmes to achieve high fidelity to a programme model within a year, with supported employment exhibiting a relatively high success rate in comparison with some other practices such as integrated dual disorders treatment. ¹¹

In the case of the programmes that the SWAN study evaluated, one may wonder whether tighter contractual arrangements and/or oversight might not have prevented what appears to have been a singular underprovision of services. It seems rather likely that the employment workers gave far too much of their time to other tasks.

The authors also acknowledge that the not-for-profit non-governmental employment agency that provided the employment specialists was not structurally or managerially integrated with the community mental health teams. They invoke this as an explanation for a less than optimal degree of day-to-day integration between clinical and IPS services – a weakness that lowered the scores of both programmes on the fidelity scale. Commissioning guidance for vocational services in the UK allows for IPS services to be delivered by the voluntary sector as well as by the National Health Service and other providers, ¹ but ensuring that employment specialists are viewed as fully fledged members of the clinical team, and co-located with them for at least part of the week, would surely help to achieve the degree of integration required for good outcomes. ¹³ Such integration would also help prevent the division of attention of the employment workers that appears to have seriously undermined the present attempt to implement IPS.

It would be unfortunate if this study were interpreted as a reason for inaction by people who, for various reasons, are reluctant to implement IPS programmes. There is now too much evidence from too many countries and contexts to doubt that IPS can be implemented successfully in the UK or probably almost any other industrialised country. Rather, this study should be interpreted as a challenge to those who wish to implement such programmes (or for that matter other complex evidence-based practices) to ensure that the conditions needed for their successful implementation are put into place.