Quantifying the contributions of three types of information to the prediction of criminal conviction using the receiver operating characteristic

Abstract

Background

Quantifying the contributions that different types of information make to the accurate prediction of offending offers the prospects of improved practice and better use of resources.

Aims

To quantify the contributions made by three types of information – demographic data alone, demographic and criminal record and demographic, criminal record and legal class of disorder – to the prediction of criminal conviction in patients.

Method

All 425 patients discharged from the three special (high secure) hospitals in England and Wales over 2 years were followed for 10.5 years. The contribution of each type of information was described in terms of the area under the receiver operating characteristic curve (AUC) and the number needed to detain (NND).

Results

The AUC values using the three types of information were 0.66, 0.72 and 0.73 respectively. Prediction based on the full model using an optimal probability cut-off implies an NND of 2. The AUCs for serious offences were 0.67, 0.69 and 0.75 respectively.

Conclusions

For long-term prediction of conviction on any charge, information on legal class adds little to the accuracy of predictions made using only a patient's age, gender and criminal record. In the prediction of serious offences alone the contribution of legal class is significant.

Age, gender and past offending are the most frequently cited correlates of criminal conviction (Bonta et al, 1998). The correlations appear to be independent of time at risk (Mossman, 1994; Buchanan & Leese, 2001). The relative contribution of these and other types of information to the successful prediction of criminal conviction in psychiatric populations has not been quantified (Mulvey, 1994). Nor is it known whether those contributions are the same irrespective of the type of offence sought to be predicted. With the partial exceptions of sexual offenders and those who commit fraud (Stander et al, 1989), however, most non-psychiatric repeat offenders do not ‘specialise’ (Farrington, 1997). This suggests that the correlates of being convicted more than once should not vary with type of offence.

We aimed to quantify the contributions made by three types of information to the prediction of criminal conviction in patients leaving high secure psychiatric care. We examined three hypotheses: first, that data relating to past offending would contribute the most to predictive accuracy; second, that this effect would be independent of the period covered by the prediction; and third, that this effect would be independent of the nature of the offence being predicted.

METHOD

Case ascertainment and operational definitions

The sample comprised all 425 patients discharged from the three special (high secure) hospitals in England and Wales between 1 January 1982 and 31 December 1983. Records of each patient's criminal convictions were obtained from the Home Office. These records covered the individual's convictions during the calendar year of discharge, during which each had been out of special hospital care for an average of 0.5 years, and for each successive year to 10.5 years. The records also covered each individual's criminal career prior to discharge. All other data were collected from the medical records departments of the three hospitals. Details of the members of the sample and of their subsequent criminal conviction have been published (Buchanan, 1998).

The Mental Health Act 1983 requires, for detention exceeding 28 days, the allocation of a patient's mental disorder to one or more of four classes: mental illness, psychopathic disorder, mental impairment and severe mental impairment. The legal and administrative nature of these classes distinguishes them from clinical diagnoses. Many patients have more than one diagnosis, not all of which are reflected in the legal class they are assigned. Significantly also, a patient's legal classification does not usually change as his or her condition fluctuates. Legal classes nevertheless contain diagnostically distinct groups of patients. The most recent survey of all three special hospitals found an 89% ‘overlap’ (and a kappa coefficient of 0.8) between legal class and clinical diagnosis (Taylor et al, 1998). Thus, 85% of special hospital patients classified as mentally ill meet ICD–10 (World Health Organization, 1992) criteria for psychosis and only 3% of patients who meet those criteria do not carry a mental illness classification (Taylor et al, 1998). Ninety-three per cent of men in Broadmoor special hospital and 97% of women in all three special hospitals who are classified as having a psychopathic disorder have one or more Axis II diagnoses under DSM–III (American Psychiatric Association, 1980). The mean numbers of DSM–III Axis II categories for those classified as having psychopathic disorder are 2.7 for men and 3.7 for women (Coid, 1992) and the most prevalent categories are antisocial, borderline and narcissistic (Reiss et al, 1999). All special hospital patients with ICD–10 severe learning difficulties have been found to be detained under severe mental impairment and 88% of those with mild to moderate learning disabilities are detained under mental impairment (Taylor et al, 1998).

The first hypothesis was examined using any criminal conviction recorded within 10.5 years of discharge as the dependent variable. To investigate the second and third hypotheses the dependent variable was changed, first to reflect convictions received at different points over the 10.5 years and second to include only conviction on a serious charge. ‘Serious’ was operationally defined to comprise murder, attempted murder, threat or conspiracy to murder, manslaughter, wounding, grievous bodily harm, actual bodily harm (multiple), child stealing, buggery, attempted buggery, rape, indecent assault, incest, gross indecency with children, robbery, kidnapping, aggravated burglary and arson. Buggery, attempted buggery, incest and gross indecency with children do not require non-consensual contact. They were included because at the time the members of the sample were convicted they were usually charged in England and Wales in cases of child abuse where the necessity of the victim giving evidence was sought to be avoided.

Characteristics of the sample

Of the 425 patients in the sample, 349 (82%) were men, and the mean age on discharge was 38 years (range 18–75, s.d.=10.9). Further data were missing for one patient. The Mental Health Act 1983 allows a patient to be placed in more than one class. Only 24 (6%) of the sample had received a dual classification, however, suggesting that legal class had been allocated to reflect the primary diagnosis rather than the full range of the patient's psychopathology. One hundred and ninety (45%) had been classified as mentally ill, 141 (33%) as having psychopathic disorder, 72 (17%) as mentally impaired and 45 (11%) as severely mentally impaired.

Patients had been convicted of a mean of 4.6 and median of 2 offences (range 0–38, interquartile range 1–6) at the time of their discharge. On discharge, 281 (66%) went to be patients in other hospitals, 122 (29%) went home or to supported accommodation and 22 (5%) went back to prison. For patients categorised as mentally ill, the proportion becoming patients elsewhere was 68% and the proportion going home or to supported accommodation was 24%. For those categorised as having a psychopathic disorder these proportions were 60% and 36% respectively and for patients categorised as mentally impaired they were 76% and 23%. Eighty-seven (20%) had been convicted of an offence within 2.5 years of discharge, 118 (28%) within 6.5 years and 134 (32%) within 10.5 years. For patients with mental illness these proportions were 17%, 23% and 29% respectively; for patients with personality disorder they were 29%, 40% and 44% and for those with mental impairment they were 15%, 20% and 25%. When other variables were controlled for, the correlates of conviction for this sample were age, number of convictions at discharge and legal class of psychopathic disorder. Prior to controlling for other variables, correlates included additionally gender and destination on discharge (Buchanan, 1998).

Statistical procedure

The variables and the order in which they were entered were chosen to place recognised risk factors for conviction (Bonta et al, 1998; Monahan et al, 2001) in a sequence that reflected the resources required to obtain the information that they comprise. Logistic regression equations were fitted to the data with each of the outcomes as dependent variables. The equations used as independent variables: first, age and gender; then, in addition, the number of convictions the patient had received prior to leaving special hospital care; and, third, in addition, each patient's legal classification. The change in likelihood at each of the three stages was used to assess the statistical significance of the new information added (twice the change in log likelihood having an approximate chi-squared distribution with degrees of freedom equal to the change in the number of parameters).

The predicted probabilities of each patient being convicted were then used to generate the coordinates of a receiver operating characteristic curve. Receiver operating characteristic analyses describe the accuracy of a procedure for classification. The curves are generated by plotting sensitivity against 1 —specificity at different thresholds. When used in conjunction with logistic regression, as here, the area under the curve (AUC) is the probability that a randomly selected case will score higher than a randomly selected non-case on the linear predictor (that is, the risk factors combined using the regression weights).

Receiver operating characteristic analyses were developed in the 1950s to describe the performance of radar (Altman & Bland, 1994). They have been used to describe the independent contributions of Gleason score, prostate-specific antigen and clinical judgement in staging prostate cancer (Partin et al, 1997; Swets et al, 2000), in the evaluation of screening programmes for hypercholesterolaemia (Umans-Eckenhausen et al, 2001) and in the identification of predictors of treatment response in multiple sclerosis (Wandinger et al, 2003). They have also been used to describe the success of attempts to predict general (Rice & Harris, 1995) and sexual (Hanson & Thornton, 2000) recidivism. They have not been employed previously to describe the contributions of different types of information to the prediction of conviction in a psychiatric population.

The description of the accuracy of a procedure for classification that a receiver operating characteristic analysis provides through the AUC does not vary with the base rate and takes into account that the sensitivity and specificity of any procedure based on a quantitative measure (such as probability of conviction) will change with the threshold that the procedure uses in order to classify. We computed the areas under the receiver operating characteristic curve at each stage and bootstrapped each change in these with 1000 replications to obtain the 95% bias-corrected confidence interval (Efron & Tibshirani, 1993). Where this excluded zero the change is noted as being significant at an approximate critical level of 0.05. Sensitivity and specificity were calculated using two probability cutoffs for predicting a case as positive: the first was 0.5 and the second was the value that maximised the sum of the sensitivity and specificity. This latter approach causes the choice of cut-off to change depending on the outcome, and is ‘optimal’ (in terms of the costs of misclassification) if the cost of a false negative result is the same as that of a false positive one (Altman, 1995).

Using the sample prevalence, the number needed to detain (NND; Buchanan & Leese, 2001) was then calculated for the optimal cut-off. The NND is the number of people who would have to be detained in order to prevent one person from being convicted. It is the inverse of positive predictive value and, unlike the AUC, is a base-rate sensitive measure that can only be calculated when the prevalence of the behaviour to be prevented is known. Also unlike the AUC, however, it describes the practical consequences of using a prediction technique to detain those thought likely to offend. Because the NND is sensitive to the rate of the behaviour to be prevented it varies with the length of time – in this case 2.5 years, 6.5 years or 10.5 years – over which a prediction is made.

Destination on discharge is associated with conviction in this sample (Buchanan, 1998) and patients categorised as having a psychopathic disorder were more likely to be discharged to unsupported settings. The final models for 10.5 years were refitted including destination as an independent variable in case this could have been a confounding factor. Two sensitivity analyses were also conducted. In the first, the AUC and other measures of predictive ability were recalculated using jack-knifing. Jack-knifing is a method of cross-validation in which cases are left out one by one, the model re-estimated with each omission and the new model used to predict the omitted case. It was used to estimate the loss of predictive accuracy that would result from using the same prediction technique with a different sample. In the second sensitivity analysis the effect of dual classification was examined by treating legal class as a single, mutually exclusive, categorical variable. Finally, we examined whether the relative contributions of the different types of information derived from the sequence in which information was added to the model.

RESULTS

The parameters of three regression models using any conviction within 10.5 years of discharge as the dependent variable are shown in Table 1. The incremental value, in terms of the area under the receiver operating characteristic curve, of adding different types of information is shown in Table 2 and in Figs 1 and 2. Demographic information produces a significant improvement over chance in the ability to predict conviction in general. Adding data from the patients’ criminal records produces a further improvement over chance that is significant both with respect to chance and with respect to the level of accuracy achieved using demographic data alone. Legal class does not significantly add to the ability to predict.

Fig. 1 Receiver operating characteristic curves: predictions of any offence within 10.5 years. AG, age and gender; AG+C, age, gender and prior conviction; AG+C+D, age, gender, prior conviction and legal class of disorder.

Fig. 2 Receiver operating characteristic curves: prediction of a serious offence within 10.5 years. AG, age and gender; AG+C, age, gender and prior convictions; AG+C+D, age, gender, prior convictions and legal class of disorder.

Table 1 Results of entering three types of data in three logistic regressions: age and gender; age, gender and prior convictions; and age, gender, prior convictions and legal class of mental disorder. The dependent variable is conviction of any offence within 10.5 years of discharge from special (high secure) hospital

Variable	OR	95% CI	P 1
Age and gender
Gender2	0.413	0.219-0.777	0.006
Age3	0.954	0.933-0.975	< 0.001
Age, gender and prior convictions
Gender	0.539	0.282-1.030	0.062
Age	0.958	0.937-0.981	< 0.001
Number of prior offences	1.092	1.051-1.134	< 0.001
Age, gender, prior convictions and legal class4
Gender	0.555	0.287-1.077	0.082
Age	0.963	0.941-0.986	0.002
Number of prior offences	1.091	1.049-1.134	< 0.001
Mental illness	1.106	0.400-3.060	0.846
Personality disorder	1.888	0.702-5.081	0.208
Mental impairment	1.132	0.514-3.377	0.566

Table 2 Predictive accuracy of three regression equations – age and gender (AG), age, gender and prior convictions (AG+C) and age, gender, prior convictions and legal class of disorder (AG+C+D) – measured in terms of the log likelihood ratio, the area under the receiver operating characteristic curve and number needed to detain

	Likelihood analysis		ROC analysis			Properties in sample (cut-off P=0.5)		Properties in sample (cut-off based on sum of sensitivity and specificity)
Outcome (prevalence)	-2 Log L1	P (increase)	AUC	Increase in AUC	P 2	Sensitivity (%)	Specificity (%)	Sensitivity (%)	Specificity (%)	NND3
Any offence 10.5 years (32%)
Constant	528		0.5
AG	500	<0.001	0.660	0.160	<0.05	8	96	70	55	2.4
AG+C	476	<0.001	0.723	0.062	<0.05	23	95	73	64	2.1
AG+C+D	472	0.169	0.730	0.007	NS	26	93	70	71	2.0
Any offence 6.5 years (28%)
Constant	502		0.5
AG	476	<0.001	0.653	0.153	<0.05	1	100	78	44	2.8
AG+C	452	<0.001	0.726	0.058	<0.05	21	96	66	70	2.2
AG+C+D	446	0.118	0.735	0.010	NS	19	94	70	71	2.1
Any offence 2.5 years (20%)
Constant	430		0.5
AG	416	<0.001	0.629	0.129	<0.05	0	100	63	55	2.8
AG+C	402	<0.001	0.680	0.051	<0.05	5	98	63	67	2.2
AG+C+D	398	0.294	0.691	0.012	NS	6	99	65	65	2.1
Serious offence 10.5 years (14%)
Constant	338		0.5
AG	320	<0.001	0.667	0.167	<0.05	0	100	79	48	5.1
AG+C	316	0.025	0.690	0.023	NS	0	100	76	53	4.9
AG+C+D	294	<0.001	0.750	0.060	<0.05	3	100	59	78	3.3

The NND shows that, applied to this population where the prevalence of any conviction in 10.5 years is 32% and using the optimal cut-off probability (in this instance of 0.30), the model including all three types of information would result in the detention of two people in order to prevent one conviction.

Table 2 also shows the results of testing the second and third hypotheses. The time over which predictions are made does not appear to affect the relative importance of different types of information to the prediction of conviction. The pattern of incremental improvement from ‘age and gender’ through ‘age, gender and prior convictions’ to ‘age, gender, prior convictions and legal class’ is maintained when the dependent variable is changed to reflect conviction by 2.5 years and 6.5 years.

For conviction on a serious charge, the addition of criminal record data to age and gender, although significant in terms of the likelihood analysis, does not add significantly to the AUC. However, in contrast to the situation for all offences, the subsequent addition of information relating to legal class does add to the ability to predict. This difference in findings for serious offences compared with all offences is evident from comparing Figs 1 and 2. Further examination of the data showed that the ability of legal class to predict serious offences could be explained by a correlation between psychopathic disorder and subsequent sexual offending (child stealing, buggery, attempted buggery, rape, indecent assault, incest, gross indecency with children; odds ratio compared with those not classed under psychopathic disorder 4.09, P=0.047).

The NND indicates that, applied to this population where the prevalence of conviction on a serious charge in 10.5 years is 14% and using the optimal probability cut-off (in this instance 0.20), the model would result in the detention of between three and four people to prevent one serious conviction.

Destination after discharge was also examined as a potential explanatory variable, along with demographic information, prior convictions and clinical information, in the 10.5-year models for predicting any offence and serious offences. Those going home or to supported accommodation were more likely to commit an offence than those going as patients to other hospitals (OR=2.97, P < 0.001, for any offence; OR=2.03, P=0.021, for serious offences). However, as had been the case when destination was not controlled for, adding clinically derived information improved the accuracy of predictions of serious offences (AUC 0.720 increasing to 0.768) but not of offences in general (AUC 0.762 increasing to 0.766).

Two sensitivity analyses were performed. In the first the predictions from the 10.5-year models were jack-knifed. There was negligible change (1% change in each of sensitivity and specificity for cut-off probability 0.5). In the second, legal class was converted into a single, mutually exclusive, categorical variable. The areas under the curve for the two 10.5-year models did not change significantly.

The relative contribution of each type of information did not depend on the sequence in which information was added to the model. For all offences, prior conviction and demographic data continued to be significantly associated with conviction when they were added after legal class (not shown). For serious offences alone, legal class when added to demographic information is significant at P=0.001. The AUC for these two types of information is 0.74. Number of prior convictions when added to this model is significant at P=0.02 but increases the AUC by only 0.01 to 0.75 (NS at P=0.05).

DISCUSSION

Limitations of the method

The method that we have used suggests at least four qualifications to the results that we report. First, the point of entry into the study of the members of our sample – discharge from special hospital – does not represent a rigid boundary between being at risk and not being at risk of conviction. More than half of these patients were discharged to other hospitals. Although some were convicted of acts committed in those other hospitals (Buchanan et al, 2004), the risk of conviction differs, for instance in terms of the opportunity to offend, the vulnerability of potential victims and the likelihood of an offence being prosecuted, while the person remains an in-patient. Imprisonment or readmission during the follow-up period will have reduced the time at risk for some members of the sample but not for others. We have not corrected for either of these effects. Second, although diagnosis – in particular, a diagnosis of antisocial personality disorder – is the only clinically derived variable that correlates consistently with future offending (Bonta et al, 1998), a patient's legal classification conveys only a fraction of the clinical information that can be collected in the course of an admission to psychiatric hospital. Similar analyses using more detailed clinical information may demonstrate a greater role for such information in the assessment of risk. Third, we used a long follow-up period. Although our subsidiary analyses using shorter follow-up periods failed to demonstrate a greater role for clinically derived information, none covered a period of less than 2.5 years. It is possible that the relative contributions of different types of information are different when predictions are sought over days or weeks. Finally, criminal conviction is only one of the proxy measures available for offending behaviour. Although there is little evidence that the correlates of offending vary according to which proxy is used (see Farrington, 1997), it is possible that a more sensitive, less specific measure such as arrest would produce different results.

This study and previous research

We have demonstrated the applicability of a statistical approach used elsewhere in medicine to the quantification of the contributions made by different kinds of information in the prediction of criminal conviction. The ability of the approach to quantify the relative contributions of those variables does not appear to be dependent on the sequence in which information is added to the regression equations. The results of jack-knifing suggest that the approach will be robust when used on other similar samples. The results nevertheless require replication using different samples and different variables, including variables that relate to a patient's clinical condition.

Our finding that demographic information and data from an individual's criminal record make significant contributions to the prediction of conviction on any charge, whereas legal class does not add to the accuracy that can be achieved using these two sources of information alone, is consistent with reviews suggesting that the long-term predictors of recidivism are the same for offenders with or without a psychiatric disorder (Bonta et al, 1998). That this remains the case regardless of the period over which predictions are made (when that period is measured in years) is consistent with earlier work showing that the overall accuracy of predictions is independent of time at risk (Mossman, 1994; Buchanan & Leese, 2001). The finding that legal class is more important in respect of serious convictions suggests, contrary to what had been thought to be the case, that predictors operate differently according to offence type.

For serious convictions the NND is higher than that for all convictions because the prevalence of serious convictions is lower and NND is a base-rate sensitive measure. The probability cut-off was chosen in a neutral way (to maximise the sum of sensitivity and specificity) and other valid choices could lead to lower values of NND, albeit at the expense of a higher false negative rate. It is possible that more sophisticated statistical methods, for example classification and regression tree analysis, would permit more accurate predictions to be made. Such methods demand substantially larger data-sets than logistic regression, however, in order to obtain robust predictors (Thomas et al, 2005).

Our AUC values should be compared with those reported elsewhere only with caution. The logistic model used for prediction has not been validated by applying it to a different sample from that used to estimate the parameters. The results of jack-knifing suggest that, at least for similar samples, the consequent loss of predictive accuracy would not be great, probably because in this case the sample size is reasonably large in comparison with the number of predictors. Such validation nevertheless invariably produces some ‘shrinkage’ in the AUC.

Mossman (1994) described a mean AUC for all prediction techniques of 0.78 and areas of 0.71 and 0.67 for validated actuarial and clinical approaches respectively. We report AUC values for any offence and serious offences of 0.73 and 0.75 respectively. The regression equations in this study used fewer actuarial and clinical data than previous studies. With the above reservation, therefore, the results suggest that when long-term predictions of conviction on any charge are sought, the incremental value of information beyond that contained in basic demographic data and a person's criminal record may be limited. The results suggest also that the clinical information generating a legal classification of psychopathic disorder is more important, and criminal record less important, when only serious offences are sought to be predicted. We are not aware of a previous quantitative description of such an effect but the finding is consistent with suggestions that criminal record is a more potent predictor of general recidivism than of more serious crimes (Bonta et al, 1998). It may be that a diagnosis of personality disorder, and hence a classification of psychopathic disorder, is operating as a proxy for histories of violence and other criminal behaviour of which the clinical team is aware but of which the person has not been convicted, and that this effect is more pronounced when the offence is serious. A second possibility is that the legal class ‘psychopathic disorder’ is operating as a proxy for ‘psychopathy’ (Hare, 1970) and that psychopathy is independently associated with conviction after discharge (Harris et al, 1991). It is unclear from previous research, however, that such an effect would be limited to serious offending (Cooke et al, 2001; Stadtland et al, 2005).

A third possibility is that adding the legal class ‘psychopathic disorder’ permits the prediction model to benefit from the reduced heterogeneity of offending by people convicted of sexual offences (Hanson & Bussière, 1998). Our method classified the majority of sexual offences as serious (Buchanan, 1998) and any ‘breeding true’ of sexual offending therefore increased the rate of serious offending disproportionately. Further inspection of our data suggested that this was the most likely explanation. Patients classed as having a psychopathic disorder were more likely than other patients to have been convicted of a serious sexual offence prior to discharge (psychopathic disorder 30%, mental illness 7.9%, mental impairment 21%, severe mental impairment 2%).

Assessment and resources

The quality of any risk assessment is limited by the availability of accurate information (Holloway, 1997). The amount and quality of the information that can be obtained is limited, in turn, by resources. Some information required by actuarial techniques, such as age and gender, is readily available. Other data, such as those contained in a criminal record, are easy to obtain in some cases but not in others. Information such as diagnosis requires more detailed – and time-consuming – assessment. These results suggest that screening patients for a history of violent behaviour may be as effective as more complicated and resource-intensive assessments of the risk of future violence, at least where all types of conviction are sought to be prevented. This seems especially likely if, as others have suggested, patients themselves will report most of the violence that can be ascertained by other means (Steadman et al, 1998).

Implications for future research

Special hospital patients in England and Wales are atypical of psychiatric patients. Further research using this approach should examine whether the relative contributions of different types of information are the same for different samples. It should also use the full range of actuarial and clinical variables that have been shown to be correlated with conviction. Of the clinical data not examined here, a history of substance misuse seems the most obvious candidate to assist in the assessment of risk (Harris & Rice, 1997; Steadman et al, 1998). Not all aspects of a person's criminal record will be of equal importance, and age at first conviction (for instance) may be a more powerful predictor than total number of convictions received.

Clinical Implications and Limitations

CLINICAL IMPLICATIONS

1. ▪ The level of accuracy in predicting conviction on any charge that can be achieved using socio-demographic and criminal record data alone appears to approach that reported using more sophisticated and resource-intensive methods.

2. ▪ In predicting conviction on any charge, such clinical information as is reflected in the Mental Health Act 1983 class of disorder adds little to the accuracy that can be achieved using socio-demographic and criminal record data alone.

3. ▪ Clinically derived data appear to add significantly to the accuracy of predictions when the convictions sought to be predicted are serious.

LIMITATIONS

1. ▪ Patients discharged from special (high secure) hospital are atypical.

2. ▪ The study used a limited amount of the information that is often available.

3. ▪ The study did not examine conviction over periods shorter than 2.5 years.