Letter to the Editor: Suicide as a complex classification problem: machine learning and related techniques can advance suicide prediction - a reply to Roaldset (2016)

We thank Dr Roaldset for his thoughtful comments on our meta-analysis, and appreciate the opportunity to discuss the important issue raised in Roaldset (2016). In his letter, Roaldset proposes an explanation for the poor predictive ability of prior self-injurious thoughts and behaviors (SITBs). Specifically, he hypothesized that when an individual is known to have a history of prior SITBs, other individuals (e.g. healthcare providers, researchers, family, friends) will invariably intervene to prevent future suicidal thoughts and behaviors. This is a reasonable and interesting potential explanation that we also considered upon obtaining our results. However, we were unable to find any empirical evidence to support this possibility; instead, we found several lines of evidence that led us to conclude that this explanation was unlikely.

First, large cross-national, population-based studies have found that most individuals who engage in SITBs do not receive treatment, with fewer than 40% of suicidal individuals worldwide receiving any form of intervention (Bruffaerts et al. 2011). The most common reason for not seeking treatment was low perceived need for care, followed by attitudinal (e.g. desire to handle one's problems) and structural barriers (e.g. financial concerns; Bruffaerts et al. 2011). Second, treatment usage has increased among individuals who engage in SITBs in recent decades; despite this, rates of suicidal thoughts and behaviors have remained virtually unchanged (Kessler et al. 2005). Third, existing evidence indicates that prior psychiatric treatment is associated with increased (rather than decreased) rates of future suicidal thoughts and behaviors (e.g. Dahlsgaard et al. 1998; Qin & Nordentoft, 2005). Aggregating across all existing longitudinal studies, our recent meta-analysis found that prior psychiatric treatment was the single strongest predictor of suicide death, and among the top predictors of suicide ideation and attempt (J. C. Franklin et al. unpublished data). These unfortunate patterns are inconsistent with the idea that prior SITBs are poor predictors because individuals with a SITB history commonly receive effective treatment or care.

We suggest an alternative interpretation of our findings. We hypothesize that prior SITBs are poor longitudinal predictors (i.e. risk factors) of future suicidal thoughts and behaviors because they have typically been considered in isolation. The processes that lead to suicidal thoughts and behaviors are likely highly complex. As such, any risk factor considered in isolation – even a relatively strong one like prior SITBs – will be an inaccurate predictor. For instance, the lifetime risk of suicide death for individuals with a mood disorder is 4.0% (Bostwick & Pankratz, 2000). This means that 96% of individuals with a mood disorder will not die by suicide. Although mood disorders may play an important role in suicide, predicting suicide death based on mood disorder history alone would be extremely inaccurate.

We reason that the prediction of suicidal thoughts and behaviors is a complex classification problem that will require the simultaneous consideration of tens or hundreds of risk factors. Classification is the process assigning data points (e.g. people) to one or more classes (e.g. will not die by suicide; will die by suicide). Classification is usually accomplished by the development of an algorithm, which is a set of rules or operations. Classification problems and their corresponding algorithms can range from simple to complex. Simple classification problems can be solved with algorithms that include a small number of rules or operations. For instance, classifying adults on the basis of biological sex is a relatively simple problem. An algorithm that considers a small number of factors (e.g. height, testosterone, hair length, voice pitch) can produce accurate classification. The traditional statistical techniques of psychology and psychiatry (e.g. general linear models) are capable of modeling such simple algorithms.

Highly complex classification problems require algorithms that model complex relationships among a large number of factors. For example, Internet search queries represent highly complex classification problems. With nearly one billion active websites and several billion active webpages, identifying which webpage matches an individual's search query is extremely difficult. To achieve this, search engines construct algorithms that integrate hundreds of unique factors. We posit that predicting future suicidal thoughts and behaviors will be at least as difficult as matching a user's search query to a relevant webpage. A search engine that would rely on a single factor alone would be highly ineffective; likewise, as demonstrated by Ribeiro et al. (2016), prediction of suicide ideation, attempt, and death using a single factor also is highly inaccurate. This was also demonstrated by Bentley et al. (2016) in a meta-analysis of anxiety disorders and symptoms as risk factors for suicidal thoughts and behaviors. Similarly, prediction was uniformly weak across individual risk factors for non-suicidal self-injury in a recent meta-analysis by Fox et al. (2015). Accordingly, we propose that studies focused on suicidal thought and behavior prediction should prioritize the development of risk algorithms over risk factors.

Traditional statistical techniques are not well-suited to model such complex algorithms. These techniques require an a priori algorithm (e.g. setting what factors should be included in the model and the specific relationships among these factors) and only consider a limited number of potential types of relationships among factors. In contrast, machine learning and related techniques are optimally suited to address complex classification problems. These methods consider highly complex relationships among a very large number of potential factors to determine the optimal classification algorithm. Applying this approach to suicide prediction has already shown promise. Recently, Kessler et al. (2015) applied machine learning methods to a large sample of Army soldiers, which yielded an area under the curve (AUC) of 0.85 – a figure considerably stronger than what the literature has been able to produce over the last five decades (i.e. AUC = 0.56; J. C. Franklin et al. unpublished data).

We thank Dr Roaldset once again for his thoughtful comments and we are grateful for the opportunity to further discuss the important issue he raised. We hope that our discussion of suicide risk as a complex classification problem and the potential of using machine learning to develop risk algorithms provides fruitful future directions for suicide prediction research.