Predictive properties of risk assessment instruments following self-harm

Summary

This month's BJPsych publishes two important studies concerned with the use of risk assessment scales after self-harm, one a systematic review and the other a multicentre cohort study. We agree with the authors: that each study adds weight to the existing evidence that points towards avoiding the use of such scales in clinical practice.

Background

When it comes to the clinical care of people who have attended the general hospital because of self-harm, are routine risk assessment scales on their way out? Last year, in an editorial right here in the BJPsych, Mulder and colleagues wrote by no means the first obituary for risk assessment scales when they were asked, rather as we have been, to comment on a newly published paper. ¹ They were weighing up a systematic review that had spun out of the meticulous compiling and analysis that the National Institute for Health and Care Excellence (NICE) does so well. That review showed that none of the various risk factors (such as gender, poor health or history of psychiatric contact), nor the three risk scales for which they could find data (the Beck Hopelessness Scale, the Suicide Intent Scale and the Scale for Suicide Ideation), are of any practical value to the assessing clinician when it comes to predicting subsequent suicide among people who have attended hospital following self-harm. ² Chan et al said that ‘The idea of risk assessment as risk prediction is a fallacy and should be recognised as such’. ² The editorial writers went one step further: ‘Our current preoccupation with risk prediction has the potential to harm patients, clinicians and the organisations in which they work [creating] a sense of unease among clinicians and a culture of blame when things go wrong’. ¹

Mounting evidence of the poor predictive power of risk scales

In this month's BJPsych are two studies that take another look at the matter: a large prospective cohort analytic study from the UK, ³ and an international collaborative systematic review with meta-analysis. ⁴ Again, both studies focus on psychosocial risk scales, although rather more of them – 7 in the cohort study, and 39 in the systematic review (from 53 separate primary studies). The outcomes include repeated non-fatal self-harm in both the present studies, with suicide an additional outcome in the review. The cohort study deals only with prediction following an episode of non-fatal self-harm whereas the systematic review also examines the use of these scales among other psychiatric patients. Because we already know, as above, that the notion of there being some clinical value from this kind of risk assessment is a fallacy, it seems inevitable that the new expanded research would draw similar conclusions.

Within the two new research reports are many findings that confirm the clinical futility of trying to use scales for the prediction of suicide or repeat self-harm. For example, the psychosocial instruments show only modest or poor positive predictive values – 39% for self-harm plus suicide (only 28% and 4% respectively for self-harm and suicide, if examined separately). ⁴ Positive predictive value is widely regarded as the measure with the most straightforward clinical meaning – the proportion of the patients who have tested positively (scoring above a threshold) who go on to experience the outcome. Plainly, a 4% predictive value means that 96% of people identified as at increased risk would not die by suicide in the follow-up period. The predictive value for non-fatal repetition, approaching 40% when combined with suicide, looks considerably better but there is something illusory about its apparent benefits when we reflect that, regardless of any testing, around 20–30% of these patients will repeat; positive predictive value is strongly associated with the incidence of the outcome event, which is rare when the event is suicide but common when it is non-fatal repetition.

The relation between positive predictive value and outcome incidence is a further problem, acknowledged by the authors, when pooling the predictive values in the meta-analysis. Each primary prediction study in the review has its separate and differing case-mix and follow-up duration, thereby rendering the pooled predictive value one that is derived from widely differing outcome incidences. The pooling of these values in a validity meta-analysis is much more of a methodological problem than it is in clinical trial meta-analysis. In a review of trials, the main finding for each trial is a comparison between two trial arms that have a common baseline outcome incidence, although those outcomes may differ widely between one trial and another. Pooling the comparison of the two arms of each trial is not invalidated by the finding of a range of baseline outcomes. When it comes to the pooling of predictive values in a validity meta-analysis, however, we find that some studies researched subgroups of patients for whom there was a high outcome incidence, and some undertook long follow-up periods, together leading to pooled positive predictive values that are likely to be substantially higher than the performance level that would be expected in regular clinical practice. ⁴

If the systematic review looks unsupportive of risk-scaling, the cohort study throws an even more unflattering light on the predictive power of risk scales in self-harm. ³ In five large English teaching hospitals, patients referred to multidisciplinary liaison psychiatry services for psychosocial assessment after self-harm were administered a structured assessment that contained the questions that make up five well-known named assessment scales – many but not all of the items were ones routinely asked but some additional questions needed to be included in the research assessment interview. They also added a clinician-rated global evaluation scale and a patient-rated version. In total, 514 patient-episodes of self-harm were assessed in this way across the five centres – each patient contributing to the validity appraisal of each of the seven scales – and each patient was followed up, using the local hospital databases, for 6 months to identify whether he or she repeated self-harm. At 30% repetition in 6 months, the positive predictive values, using established cut-off points, varied from a meagre 13% (the Modified SAD PERSONS scale) to 47% for the clinician's global scale. So, the scales' predictive values ranged from worse-than-useless to providing some modest predictive advantage, although the clinician's rating was the scale that offered the best forecast (and the patient's own global rating was the next best).

The authors went on to use a technique familiar to these kinds of evaluations: they recalculated cut-off points post hoc. That is, they used the findings to determine the best possible cut-off threshold for their particular study sample. It is important to emphasise that the revised validity is thereby one that maximises the validity metrics; if the study were repeated in another, independently assembled sample of patients (or in the real world) the scales would function a little or a lot worse. ⁵ In these ideal calculations, the self-harm prediction scales are crowded together and provide a range of positive predictive values between 33% and 47%; at the top end of this range of values they accomplish no more than we saw in the systematic review. ⁴ Another useful way of judging validity is the plotting of receiver operating characteristic (ROC) curves and the calculation of areas under the curves of competing scales; the seven scales here show poor-to-worthless performance. In short, no scale provides a reasonably accurate prediction of repetition of self-harm.

The way forward – needs assessment

What then is to be done? The research cited above and the earlier editorial ^1–4 consistently set out a clear recommendation: risk assessment scaling should not form the basis of clinical care, and the use of detailed risk assessment scales should be curtailed lest they deliver false reassurance for clinicians and managers. ¹ National (NICE) guidelines already tell us: ‘Do not use risk assessment tools and scales to predict future suicide or repetition of self-harm [or] to determine who should or should not be offered treatment or who should be discharged’ ⁶ – yet use of these tools remains widespread. ⁷ Instead, thorough psychosocial assessment is recommended after self-harm – to ‘move away from assessment models that prioritise risks at the expense of needs’ ² and to focus on the person and the particular circumstances, characteristics and meanings that seem likely to have precipitated his or her suicidal ideation and behaviour. Needs assessment is a sufficiently apt term for such an approach and it is probably much closer to what patients actually want.

Evidence, largely qualitative, about recipients' views of the psychosocial assessment process offers some straightforward messages, although what people think specifically about being in receipt of risk-scaling is not known. There are copious reports of dissatisfaction with any sense of being processed, with stock questions in particular deemed to constitute a superficial assessment, for example: ‘I could say, “how are you feeling now?” and get paid for it … in that sort of work you've got to have your own opinions and I think that's what makes the better ones that don't stick to the standard questions, yes and no answers;’ ⁸ and ‘it's like they're pre-programmed to ask these questions irrespective, you could go with your head dropping off and they'd still be asking these questions’. ⁸

It is plain that patients want a space where it is safe to be emotionally distressed while retaining some sense of privacy. ⁸ When it comes to investigations, physical or psychological, people want clear explanations of why they are being done, what is involved, what the findings and implications are, and want to discuss what further symptoms they might expect as a consequence of the self-harm and of any treatment given. ⁸ People also want to share in decision-making about their future care, with reasonable attention paid to their personal preferences. ^9,10 It is not that people who have attended hospital as a consequence of self-harm want distinct or exceptional treatment. Instead, they want the same level of clinical care that might be expected by anyone else in the emergency department or general ward, delivered with the same level of openness, warmth and respect – although accompanied by acknowledgement of their fragile emotional state. They also want assurance that they are not viewed as time-wasters or attention-seekers. ¹¹ Care delivery, including the psychosocial assessment process, needs to be sufficiently compassionate. People are relieved to have their painful mental state taken seriously and when the nurse or doctor legitimises feelings of distress, it can be a first step in dealing with the intense negative emotions that preceded the self-harm. ¹² People who have self-harmed know that there will be some routine questioning and a necessary assessment process before they can go home ¹² and they know it will be required before arrangements that they hope will be helpful can be put in place: ‘[I'm] hugely grateful that I've got the help, it's made a whole world of difference … people are phoning me, keeping me informed, my care people are coming, I know that within the next couple of weeks, I will have the support I need’. ¹²

Unfortunately, there is a substantial flaw in any recommendation concerned with improvements in the quality and focus of psycho-social assessments, revealed by a recent survey of hospital services for self-harm in 32 hospitals in England by Cooper and colleagues. ¹³ The study reveals a sad state of affairs regarding hospital services for self-harm: the median figure for the proportion of people receiving a psychosocial assessment from a mental health professional following self-harm was only 58%, and was as low as 22% at worst – worrying rates given the strong links between self-harm and subsequent suicide. This survey was a re-evaluation of the same hospitals in a similar survey almost 10 years earlier and there seemed to have been no improvement on the low levels of adequate assessment seen then, although there was some evidence of improved service structures. Further corroboration of this habitual deficiency in the services in England can be found in the present cohort study: ³ in the teaching hospitals where the research was carried out, psychosocial assessment was only received by 45% to 77% of people. It is hard to see how a focus on individual patients' needs, as recommended here and elsewhere, can be achieved when many (and in some places most) patients do not receive an assessment at all, let alone a proper evaluation of those needs.