Getting real about risk

The recent meta-analysis by Singh et al ¹ examined the proportion of violent people among those classified as high risk, known as the positive predictive value (PPV). They found that PPV is highly variable between studies and is most strongly associated with the base rate of violence in the whole risk-assessed group. They conclude that risk assessment is not a reliable indicator of absolute risk. We agree. The increased focus on the PPV of high-risk categories is a welcome development because it leads to a consideration of the number of people who might need to be assessed as high-risk for every true positive (actually violent) person. The number needed to assess is often a more relevant measure than those derived from the receiver operator curve and it clearly illustrates the lack of meaning in recent debates about the extent to which group data apply to individuals ² and the margins of error in particular risk predictions. ³ However, we believe that the debate about risk assessment now needs to move beyond abstract notions relating solely to probability. A probability after all is simply a number between 0 and 1, a number that is uninformative unless it is a probability of something specific.

Although not cited in Singh et al, we systematically examined PPV of risk categorisation after making generous assumptions about the statistical power of risk assessment. ⁴ Unlike Singh et al, our paper focused on the main factor that actually determines base rates and thus PPV - the definition of violence according to violence severity. For example, using a risk assessment instrument with a sensitivity and specificity of 80% for the detection of different outcomes, the PPV for criminally violent behaviour over a year by people with schizophrenia might be about 4% under optimal conditions, whereas the same figure for homicide would be 0.04%.

In the primary risk research, including that used by Singh et al, ¹ a wide spectrum of violent events is amalgamated into an omnibus ‘violent’ category. These events range from common assault all the way to homicide. Each of these diverse events has different base rates and consequences, with more severe violence having lower base rate but leading to greater losses.

Risk assessment in mental health should start to consider the dimension of resulting loss. In areas outside mental health, risk is not a probability but is a quantum of loss - that is why we pay our insurance premium in money, yet have little idea of the likelihood of the loss of our possessions. In our view any study that does not consider the magnitude of resulting loss should not really be thought of as a ‘risk assessment’ and more properly should be referred to as ‘probability assessment’. Although quantification of loss poses significant challenges, considering a definition of risk that includes the loss component re-emphasises two complex, important and unanswered questions. First, what actual psychiatric interventions in terms of cost/side-effects/benefits are indicated for those who are regarded as high risk, and yet should be withheld from patients classified as low risk? If the harm we consider is not severe, no costly, restrictive or intrusive treatment can be justified. If the harm considered is severe, it will also be rare. Therefore, what costly and intrusive intervention can be justified for the tiny proportion of false positives, or if the intervention is not costly or intrusive, why withhold it from low-risk patients, who will commit many of the future acts of violence? ⁵ Second, is there evidence that an overall reduction in violence can be achieved by applying this cost-effective and acceptable intervention to a group who are more likely to offend while denying it to those who as a group are less likely to offend? Will the additional resources spent on preventing violence by high-risk patients be justified in terms of harm reduction? ⁶

At the end of the recent paper, Singh et al recommend that risk assessments be provided with a qualification explaining their limitations. Here we agree as well. Perhaps it should be ‘this risk assessment provides an estimate of an uncertain probability of an unspecified event with no consideration of the consequences’.