Developing a clinically useful actuarial tool for assessing violence risk

Abstract

Background

A new actuarial method for violence risk assessment – the Iterative Classification Tree (ICT) – has become available. It has a high degree of accuracy but can be time and resource intensive to administer.

Aims

To increase the clinical utility of the ICT method by restricting the risk factors used to generate the actuarial tool to those commonly available in hospital records or capable of being routinely assessed in clinical practice.

Method

A total of 939 male and female civil psychiatric patients between 18 and 40 years old were assessed on 106 risk factors in the hospital and monitored for violence to others during the first 20 weeks after discharge.

Results

The ICT classified 72.6% of the sample as either low risk (less than half of the sample's base rate of violence) or high risk (more than twice the sample's base rate of violence).

Conclusions

A clinically useful actuarial method exists to assist in violence risk assessment.

A recent survey in the USA found that 95% of the general public believes that when a person with a mental disorder is predicted to be violent to others, legal intervention to avert the anticipated harm is justified (Pescosolido et al, 1999). The pervasiveness of such belief throughout the world helps to explain the wide and growing variety of laws, including, in-patient and out-patient civil commitment, sexual predator commitment, tort liability and employment discrimination, that call upon psychiatrists and psychologists to assess the risk of violence (Reed, 1997). A large body of research conducted in the 1970s and 1980s called into question the ability of clinicians to make accurate risk assessments of violence of the type that the public and the law demand. More recent research (Lidz et al, 1993) has been only slightly more sanguine. One repeatedly suggested way to improve the accuracy of risk assessments of violence has been to use statistical or actuarial methods to inform clinical judgement (Borum, 1996). However, where clinically relevant actuarial tools have been available, their application has been sufficiently cumbersome and time-consuming that actuarial input into the risk assessment of violence has been impractical in most real-world clinical settings (Gardner et al, 1996).

ITERATIVE CLASSIFICATION TREE METHOD

We have recently developed an actuarial tool for assessing the risk of violence by people discharged from psychiatric facilities that we believe has greater potential for clinical application than existing actuarial methods (Steadman et al, 2000). We call this tool the Iterative Classification Tree (ICT). Classification trees (Breiman et al, 1984) have long been used in a number of areas of medicine, including neurology (e.g. Levy et al, 1985). A classification tree approach to the risk assessment of violence is predicated upon an interactive and contingent model of violence: one that allows many different combinations of risk factors to classify a person as high or low risk. Whether a particular question is asked in any clinical assessment grounded in this approach depends on the answers given to each prior question. Based on a sequence established by the classification tree, a first question is asked of all persons being assessed. Contingent on the answer to that question, one or another second question is posed, and so on, until each person is classified into a category on the basis of the risk of violence. This contrasts with the usual approach to actuarial risk assessment in which a common set of questions is asked of everyone being assessed and every answer is weighted and summed to produce a score that can be used for the purposes of categorisation.

In addition to its tree-based character, our approach acknowledges the practical impossibility of adequately classifying all persons into a high or a low violence risk group. Therefore, rather than relying on the standard single threshold for distinguishing among cases, our approach employs two thresholds: one for identifying high-risk cases and one for identifying low-risk cases. We assume that inevitably there will be cases that fall between these two thresholds, cases for which any prediction scheme is incapable of making an adequate assessment of high or low risk. Based on current knowledge, the aggregate degree of risk presented by these intermediate cases cannot be distinguished statistically from the base rate of the sample as a whole.

CLINICAL UTILITY

Our first test of the ICT method (Steadman et al, 2000) focused on how well the method performed in making risk assessments of violence under ideal conditions (i.e. with few constraints on the time or resources necessary to gather risk factors). For example, the risk factor that most clearly differentiated high-risk from low-risk groups was the Hare Psychopathy Checklist: Screening Version (Hare PCL-SV; Hart et al, 1995). Given that the full Hare PCL-R requires several hours to administer - the screening version alone takes over one hour to administer - and that it has to be administered by experienced clinicians whom Hare (1998) recommended should undergo three days of specialised training, resource constraints in many non-forensic clinical settings will preclude its use.

This article applies the ICT method to the sample of patients assessed in the MacArthur Violence Risk Assessment Study (Steadman et al, 1998). Our goal is to increase the clinical utility of this actuarial method by restricting the risk factors tested to those commonly available in hospital records or capable of being assessed routinely in clinical practice.

METHOD

Subject enrolment

Admissions were sampled from acute psychiatric in-patient facilities at three sites: Western Psychiatric Institute and Clinic (Pittsburgh, PA); Western Missouri Mental Health Center (Kansas City, MO); Worcester State Hospital and the University of Massachusetts Medical Center (Worcester, MA). Selection criteria for research subjects were: civil admissions; between the ages of 18 and 40 years; English-speaking; White or African American ethnicity (or Hispanic in Worcester only); and a chart diagnosis of schizophrenia, schizophreniform disorder, schizoaffective disorder, depression, dysthymia, mania, brief reactive psychosis, delusional disorder, alcohol or drug abuse or dependence, or a personality disorder. After complete description of the study to the subjects, written informed consent was obtained.

Sample description

We approached a quota sample (to ensure representativeness across sites on gender, race, and age) of 1695 to participate. The refusal rate was 29% (n=492). The final sample given a hospital interview was 1136. Differences between the eligible admissions and the follow-up sample (n=939) are discussed in detail elsewhere (Steadman et al, 1998). Males comprised 57.3% of the sample. Ethnically, 68.7% of the sample was White, 29.1% African American and 2.2% Hispanic. The mean age was 29.9 (s.d.=6.2) years. Depression was the most frequent primary research diagnosis on the DSM-III-R Checklist (Janca & Helzer, 1990; 41.9%), followed by alcohol/drug abuse or dependence (21.8%), schizophrenia (17.0%), bipolar disorder (14.1%), personality disorder only (2.1%) and other psychotic disorder (3.1%). The proportion of all cases with a primary research diagnosis of major mental disorder that had a co-occurring diagnosis of substance abuse or dependence was as follows: depression, 49.6%; schizophrenia, 41%; bipolar disorder, 37.7%; and other psychotic disorder, 45%.

Hospital data collection

Hospital data collection was conducted in two parts: an interview by a research interviewer to obtain data on risk factors and violence; and an interview by a research clinician (PhD or MA/MSW in psychology or social work) to confirm the chart diagnosis using the DSM-III-R Checklist and to administer several clinical instruments.

The hospital data set assembled in the MacArthur Violence Risk Assessment Study consisted of 134 risk factors from four conceptual domains: dispositional or personal factors (e.g. age); historical or developmental factors (e.g. child abuse); contextual or situational factors (e.g. social networks); and clinical or symptom factors (e.g. delusions) (Steadman et al, 1994). For the present analysis, we eliminated 28 risk factors that would be the most difficult to obtain in clinical practice, restricting ourselves to the remaining 106. Two criteria were used to eliminate risk factors. The first was to eliminate information generally unavailable to mental health personnel in the context of brief hospitalisation (e.g. information in official arrest records, in distinction to self-report of prior arrests). The second was to eliminate information that required the administration of a lengthy (>12-item) instrument to obtain (e.g. a social network inventory (Estroff & Zimmer, 1994)). A list of all 134 risk factors, with their bivariate correlations with violence and with an indication of which were eliminated from these analyses, is provided in Table 1.

Table 1 Risk factors and bivariate correlations with violence in the first two follow-ups

Domain	Reference	Pearson coefficient R
Personal
Gender: male		0.08*
Age		-0.07*
Race: White		-0.12***
Verbal IQ1		-0.11**
Ever married		0.01
Hare Psychopathy Cheklist: Screening Version >121	Hart et al, 1995	0.26***
Novaco anger: behaviour	Novaco, 1994	0.16***
Novaco anger: cognitive		0.11*
Novaco anger: arousal		0.09**
Novaco anger: intensity		0.08*
Barratt impulsiveness: motor	Barratt, 1994	0.07*
Barratt impulsiveness: non-planning		0.05
Barratt impulsiveness: cognitive		0.05
Historical
Years of education		-0.11***
Socio-economic status1	Hollingshead & Redlich, 1958	0.05
Employed		-0.05
Age at first hospitalisation		-0.04
No. of prior hospitalisations		-0.03
Involuntary legal status		0.11**
Recent violent behaviour		0.14***
Adult arrest: seriousness		0.25***
Adult arrest: frequency		0.24***
Any arrest: person crime1	Official Report	0.13***
Any arrest: other crime1	Official Report	0.11***
Sexually abused before age 20		-0.03
Seriousness of abuse as child		0.14***
Frequency of abuse as child		0.12***
Father ever used drugs		0.16***
Father ever arrested		0.15***
Father ever excess drinking		0.11**
Father ever admitted to psychiatric hospital		0.02
Lived with father to age 15		-0.09**
Mother ever used drugs		0.05
Mother ever arrested		0.05
Mother ever excess drinking		0.06
Mother ever admitted to psychiatric hospital		-0.02
Lived with mother to age 15		-0.06
Parents ever fought with each other		0.06
Parents ever fought with others		0.03
Any head injury: loss of consciousness		0.10**
Any head injury: no loss of consciousness		0.06
Self-harm thoughts		0.02
Self-harm attempt		-0.03
Attempt to kill self		0.01
Contextual
Living in private residence		-0.05
Homeless		0.05
Living alone		-0.07*
Perceived stress1	Cohen et al, 1983	0.08*
Social networks	Estroff & Zimmer, 1994
No. of people in social network1		-0.02
% Mental health professionals in social network1		-0.10**
% Family in social network1		0.01
No. of negative persons in social network1		0.07*
No. of positive and material supporters1		-0.07*
Average no. of mentions per negative person1		0.06
Average no. of mentions per positive material person1		-0.03
Frequency of social network contact1		-0.03
Duration of social network contact1		0.02
Clinical
Chart antisocial personality disorder		0.19***
DSM-III-R Checklist	Janca & Helzer, 1990
Major disorder, no substance		-0.19***
Major disorder and substance		0.08*
Substance, no major disorder		0.15***
Drug or alcohol		0.18***
Drug		0.17***
Alcohol		0.14***
Schizophrenia		-0.12***
Mania		-0.04
Depression		-0.02
Other psychosis		0.00
Personality disorder only		0.02
Brief Psychiatric Rating Scale	Overall, 1988
Total score1		-0.04
Activation sub-scale1		-0.08*
Hostility sub-scale1		0.08*
Anergia sub-scale1		-0.07*
Thought disturbance sub-scale1		-0.06*
Anxiety/depression sub-scale1		0.01
Global Assessment of Functioning	American Psychiatric Association, 1989	-0.05
Activities of daily living		-0.01
Delusions	Appelbaum et al, 1999
Any delusions1		-0.06
Persecutory		-0.07*
Grandiose		-0.01
Body/mind control		-0.09**
Thought broadcasting		-0.05
Religious		-0.08*
Jealousy		-0.02
Guilt		-0.03
Somatic		-0.03
Influence on others		-0.03
Threat/control-override		-0.10**
Other		-0.04
Violent fantasies	Grisso et al, 2000
Any		0.13***
Frequent		0.13***
Recent onset		0.07*
Same target		0.03
Focus same person		0.10**
Escalating harm		0.13***
While with target		0.12***
Frequent, not escalating, not with target		-0.01
Frequent, escalating, not with target		0.09**
Frequent, not escalating, with target		0.08*
Frequent, escalating, with target		0.10**
Not frequent, not escalating, not with target		0.13***
Any hallucinations		0.02
Command hallucinations		0.06
Present at time of admission	Record Review
Substance abuse		0.14***
Paranoia		-0.09**
Delusions		-0.09**
Decompensation		-0.09**
Violence		0.09**
Hallucinations		-0.07*
Bizarre behaviour		-0.07*
Medication non-adherence		-0.07*
Aggressive (non-violent)		0.06
Anxiety		-0.05
Suicide attempt		0.05
Mania		-0.04
Personal problems		0.03
Evaluation		0.03
Other		-0.03
Medication change		-0.02
Unable to care for self		0.02
Suicide threat		-0.01
Property damage		-0.01
Court order		-0.01
Depression		-0.003
Drug use
Any drug		0.12***
Cocaine		0.11**
Alcohol		0.10**
Other		0.08*
Marijuana		0.04
Stimulants		0.04
Sedatives		0.03
Opiates		0.04
Mini-Mental State1	Folstein et al, 1975	0.02
Perceived coercion at admission1	Gardner et al, 1993	0.03

Community data collection

Twenty weeks after hospital discharge was chosen as the time frame for the analysis here because this was the period during which the prevalence of violence by patients in the community was at its highest (Steadman et al, 1998). Research interviewers attempted two follow-up interviews with enrolled patients in the community during this period, approximately 10 weeks apart. A collateral informant who knew of the patient's behaviour in the community during the follow-up period - usually, but not always, a family member - was also interviewed on the same schedule. Arrest and re-hospitalisation records provided the third source of information about the patients' behaviour in the community.

Patients and collaterals independently were asked whether the patient had been involved in any of several categories of violent behaviour in the past 10 weeks (Lidz et al, 1993). Only the most serious act for each discrete incident was coded. Violence to others was defined to include the following: acts of battery that resulted in physical injury; sexual assaults; assaultive acts that involved the use of a weapon; or threats made with a weapon in hand. (Battery that did not result in injury was defined as ‘other aggressive act’ (Steadman et al, 1998) and is not considered in the analyses reported here.) Violence reported by any of the three data sources - subject self-report, collateral report, or official records - was reviewed by a team of trained coders. Ethical and legal issues encountered in conducting this research are discussed elsewhere (Monahan et al, 1994).

Developing the classification tree

To develop the ICT model, we used CHAID (chi-squared automatic interaction detector) software (SPSS, 1993). Specifically, the CHAID algorithm was used to assess the statistical significance of the bivariate association between each of the 106 eligible risk factors and the dichotomous outcome measure - violence in the community - until the most statistically significant value of χ² was identified, with P < 0.05 a necessary condition for risk factor selection. Once a risk factor was selected, the sample was partitioned according to the values of that risk factor. This selection procedure was then repeated for each of the sample partitions, thus further partitioning the sample. The result of the partitioning process was to identify groups of cases that shared the same risk factors and that also shared the same values on the outcome measure of violence.

Iterating the classification tree

We then extended this recursive partitioning approach in an iterative fashion. That is, all subjects not classified into groups designated as either high risk or low risk in the first iteration of CHAID were pooled together and re-analysed in a second iteration of CHAID. This iterative process continued until it was not possible to classify any additional groups of subjects as either high or low risk (with no group allowed to contain fewer than 50 cases).

Choosing two cut-offs

The choice of cut-off scores for high-risk and low-risk categories must be made in the context of legal or policy values external to the methodology chosen for assessing risk. Here, for illustrative purposes, we defined any group of patients with a rate of violence that was less than half the base prevalence rate of the total sample, as in the low-risk category, and any group of patients whose rate of violence was greater than twice the base prevalence rate of the total sample, as in the high-risk category. Because the base prevalence rate of violence during the first 20 weeks after hospital discharge for the total sample was 18.7% (i.e. 18.7% of the patients committed at least one violent act during either the first or second 10-week follow-up period), this meant that the cut-off for the low-risk category was 9% violent and the cut-off for the high-risk category was 37% violent.

The ICT contained three iterations (Fig. 1). In the first iteration, the tree classified 429 of the 939 subjects (45.7%) into either the high- or the low-risk categories. In the second iteration, the tree classified as high- or low-risk 167 (32.7%) of the 510 subjects who were not classified into either high- or low-risk groups at the end of iteration 1. In the third iteration, the tree classified as high- or low-risk 86 of the 343 subjects (25.1%) who were unclassified at the end of iteration 2. At the end of iteration 3, no further groups could be classified as high- or low-risk, given the parameters of the model we had set (e.g. no group with fewer than 50 cases); 257 subjects (27.4% of the total sample) remained unclassified. The final ICT contained 15 contingent risk factors that formed 11 risk groups (four low-risk groups, accounting for 50.9% of the total sample; three high-risk groups, accounting for 21.7% of the total sample; and four unclassified risk groups, accounting for 27.4% of the sample).

Fig. 1 Clinically useful iterative classification tree. – – –, Low violence risk; ——, high violence risk; · · · ·, unclassified.

The risk factors displayed in Fig. 1 are defined as follows. Seriousness of prior arrests was a patient's self-report of the seriousness of arrests since age 15 years. Motor impulsiveness was measured from the motor sub-scale of the Barratt Impulsiveness Scale (Barratt, 1994). Father used drugs was a self-report question on whether the patient's father ever used drugs excessively. Recent violent fantasies was measured by the Schedule of Imagined Violence (Grisso et al, 2000). Major disorder without substance abuse refers to a diagnosis of any major mental disorder without any co-occurring substance abuse diagnosis, as reached by research clinicians using the DSM-III-R Checklist. Legal status was the initial status for the baseline hospitalisation, as recorded in hospital admission records. Schizophrenia was the diagnosis reached by research clinicians using the DSM-III-R Checklist. Anger reaction was measured by a short version of the Behavioural Subscale of the Novaco Anger Scale (Novaco, 1994). Employed was a self-report question regarding the patient's paid full- or part-time employment status in the two months prior to hospital admission. Recent violence was a self-report of violence in the two months prior to hospital admission. Loss of consciousness referred to a self-report of any loss of consciousness due to head injury. Parents fought was a self-report by the patient that his or her parents engaged in physical fights with one another when the patient was growing up. (A complete list of the questions comprising these risk factors is available from the first author upon request.)

Receiver operating characteristic

To assess the predictive accuracy of the actuarial model produced by this method and to facilitate further comparisons of our results with other research on violence risk assessment, we used a receiver operating characteristic (ROC) analysis (Gardner et al, 1996; Quinsey et al, 1998). The statistic used to summarise the analysis is the area under the ROC curve, which corresponds to the probability that a randomly selected violent patient will have been assessed by the risk assessment tool as higher risk than a randomly selected non-violent patient (Swets, 1988). The area under the ROC curve for the 11 risk groups presented in Fig. 1 is 0.80 (P<0.001). The distribution of cases that were violent or not violent during the follow-up as a function of the low- and high-risk cut-offs used to generate the ICT is presented in Table 2.

Table 2 Distribution of violent and non-violent cases using two thresholds

Observed	Classification
	Low <9%	Unclassified	High >37%	Total
Not violent	456	195	112	763
Violent	22	62	92	176
Total	478	257	204	939

Bootstrapping

We did not cross-validate the ICT. Cross-validation of a risk assessment model requires estimating the model on a subset of the data and validating the model on the rest. As noted by Gardner et al (1996), however, cross-validation “wastes information that ought to be used estimating the model” (p 43). For this reason, bootstrapping (Mooney & Duval, 1993) has become a widely used analytical strategy for estimating the shrinkage to be expected when a model is generalised to a sample other than the one on which it was estimated. In conducting such an analysis, 1000 bootstrapped samples were drawn from the original data set. Table 3 presents the 95% confidence intervals for each of the 11 risk groups in the ICT, in order of decreasing risk. The ranges of these intervals indicate how the ICT is likely to perform on other similar samples.

Table 3 Bootstrapped 95% confidence intervals for the ICT risk groups

Risk group	% violent in risk group	95% confidence interval
B	52.7	41.0-63.8
F	41.8	31.3-52.5
A	39.2	26.2-52.4
G	34.4	22.5-46.1
J	28.2	20.6-35.8
I	18.9	8.2-29.4
K	10.9	3.6-18.2
C	7.4	1.6-13.2
E	6.8	1.3-12.1
H	4.7	0.2-8.8
D	2.7	0.5-4.9

DISCUSSION

We have sought to increase the utility of an actuarial method for real-world clinical decision-making by applying the newly developed ICT method (Steadman et al, 2000) to a set of violence risk factors commonly available in clinical records or capable of being assessed routinely in clinical practice. We have shown that the ICT partitioned 72.6% of a sample of discharged psychiatric patients into one of two categories with regard to their risk of violence to others during the first 20 weeks after discharge. One category consisted of groups whose rates of violence were no more than half the base rate of the total patient sample (i.e. ≤9% violent). The other category consisted of groups whose rates of violence were at least twice the base rate of the total patient sample (i.e. ≥37% violent). The actually observed rates of violence in the low- and high-risk categories were 5% and 45%, respectively. The prevalence of violence within individual risk groups within the low- and high-risk categories varied from 2.7% to 52.7% (Table 3).

The ICT left 27.4% of the total sample unclassified, meaning that it could find no combination of risk factors that allowed these patients to be classified into either a low- or a high-risk group. The violence rate for the unclassified category was 24.1%.

Clinical illustrations

Illustrating the use of the ICT may be helpful. A clinician evaluating a patient's risk of violence using the ICT presented in Fig. 1. would first ask the patient about the seriousness of his or her prior arrests. If the patient stated that he or she had previously been arrested for a violent crime, the clinician would then inquire into whether the patient recently had been fantasising about being violent. If the patient responded affirmatively to this second question, he or she at that point would be placed in the high violence risk category. More specifically, the patient would be placed in risk group B, a group in which approximately 53% of the patients are expected to commit a violent act in the next several months.

If, on the other hand, the patient denied having violent fantasies, the clinician would then indicate whether the patient had a diagnosis of schizophrenia. If the patient did have such a diagnosis, he or she at that point would be placed in the low violence risk category. More specifically, the patient would be placed in risk group E, a group in which approximately 7% of the patients are expected to commit a violent act in the next several months. (For other studies finding rates of violence to be lower among patients with schizophrenia than among patients with other, primarily personality disorder, diagnoses, see: Gardner et al, 1996; Quinsey et al, 1998; Wallace et al, 1998.)

Comparative predictive accuracy

We have demonstrated here that the ICT method may be adapted for clinical use. The method does not require unavailable or costly-to-gather data in order to characterise the risk of violence. Rather, risk factors usually found in patient files, or capable of routine assessment, are all that are required for the ICT to function. The predictive accuracy of the ICT using a reduced set of 106 clinically feasible risk factors from the MacArthur Violence Risk Assessment Study (an area under the ROC curve of 0.80) is comparable to the predictive accuracy that we reported (Steadman et al, 2000) for risk assessment using the expanded set of 134 risk factors (an area under the ROC curve of 0.82).

Violence risk assessment software

Although the contingent nature of the risk factors identified in Fig. 1 may appear too intricate for use in clinical practice, the utility of the ICT model would be enhanced greatly with the aid of software. Software would facilitate the assessment of an individual patient by guiding the clinician to ask only those questions required to assess risk. We are in the process of developing such software.

Clinical Implications and Limitations

CLINICAL IMPLICATIONS

• ▪ The classification tree approach to violence risk assessment proposed here improves on traditional approaches by explicitly acknowledging that violence is an outcome reached by multiple routes.

• ▪ Employing two cut-off scores - one for identifying high-risk cases and one for identifying low-risk cases - is more realistic than attempting to classify all patients as to the risk of violence.

• ▪ A clinically useful actuarial approach for assessing the risk of violence among acute psychiatric patients now exists.

LIMITATIONS

• ▪ This is a clinical study of violence among people hospitalised for mental disorders, not an epidemiological study of violence among people with mental disorders in the general population.

• ▪ The extent to which the accuracy of the actuarial tool developed here generalises to other types of clinical setting (e.g. forensic hospitals) is unknown.

• ▪ The proposed tool is specifically designed for assessing the risk of violence; efforts to manage the risk of violence will require additional data.