Look back in anger: flaws in the retrospective evaluation of risk assessment

The team from the National Confidential Inquiry into Suicide and Homicide by People with Mental Illness recently posted a report on quality of risk assessment prior to suicide and homicide. ¹ The report describes an un-masked retrospective survey of the risk assessments found in the case notes of 42 suicide victims and 39 homicide perpetrators. The authors suggest that in about a third of cases poor risk assessment might have contributed in some way to those deaths. However, the assumptions made in the report and the interpretation of the results raise serious concerns.

First, the study did not examine whether risk assessments that were classified as inadequate were more common in the notes of suicide or homicide cases than in the notes of other comparable patients with a non-fatal outcome. In fact, there is no evidence to show that raters who are masked to the eventual outcome can correctly identify the notes of patients who have died by suicide ² or perpetrated homicide.

Second, there is no empirical evidence to suggest that risk assessment is of any use in preventing rare events such as suicide and homicide. ³ The low base rate of these events means that for every correct prediction there are inevitably a very large number of false positive predictions, reducing the possibility of arranging any practical intervention to prevent the adverse outcome. The low proportion of true positives means that any intervention that follows from a high-risk categorisation with the aim of preventing a rare outcome must be sufficiently effective and benign to warrant treating so many false positives, and must be efficient enough so as not to result in the excessive diversion of healthcare resources from low-risk patients, including the proportion of false negative categorisations. The lack of sensitivity of risk assessments means that they miss about half of all homicides ⁴ and as many as 90% of all suicides. ⁵ Hence, if there are benign life-saving interventions that are suitable for high-risk patients, they should be offered to so-called low-risk patients as well, obviating the need for a risk assessment.

Hindsight will always allow us to identify clinical decisions that might have prevented an adverse event, like the action replays of goals in football matches. However, when viewed prospectively, clinical decisions involve a level of uncertainty and the requirement that the clinician accepts a level of risk in order to respect the patient’s wishes and ration the resources that are available. Perhaps the reason that a third of risk assessments were thought to be of poor quality was because those clinicians were aware of the futility of trying to predict a rare event, and were just getting on with doing what is possible, which is a comprehensive assessment of the individual patient’s treatment needs, followed by ethical and compassionate evidence-based treatment of every patient.