In 2019, over 6000 Veterans died from suicide in the US, a rate that is, when adjusting for age and sex, 57% higher than civilian adults' suicide rate (OMHSP, Reference OMHSP2021). While the US Department of Veterans Affairs (VA) has made important suicide prevention contributions, these efforts have primarily targeted high-risk Veterans with documented suicide risk, including suicidal ideation, prior suicide attempts, and other flagged concerns (Matarazzo et al., Reference Matarazzo, Eagan, Landes, Mina, Clark, Gerard and Reger2023; McCarthy et al., Reference McCarthy, Bossarte, Katz, Thompson, Kemp, Hannemann and Schoenbaum2015). More than 90% of VA patients that go on to die by suicide (Kessler et al., Reference Kessler, Hwang, Hoffmire, McCarthy, Petukhova, Rosellini and Bossarte2017), however, do not meet these criteria and therefore do not fall into this high-risk tier, nor receive targeted suicide prevention services (McCarthy et al., Reference McCarthy, Cooper, Dent, Eagan, Matarazzo, Hannemann and Katz2021).
Predicting suicide remains notoriously challenging (Nock, Ramirez, & Rankin, Reference Nock, Ramirez and Rankin2019), constrained by the relatively low rates of suicide and the diversity of symptom typologies. Given this difficulty, the VA's suicide prevention strategy pragmatically focuses on patients with the highest likelihood of dying by suicide (Kessler et al., Reference Kessler, Hwang, Hoffmire, McCarthy, Petukhova, Rosellini and Bossarte2017). This strategy has led to a range of risk classification innovations, including Recovery Engagement and Coordination for Health – Veterans Enhanced Treatment (REACH-VET: Cannizzaro, Reference Cannizzaro2017; Kessler et al., Reference Kessler, Hwang, Hoffmire, McCarthy, Petukhova, Rosellini and Bossarte2017), a suicide risk prediction algorithm that automatically evaluates all VA patients for high-risk status. REACH-VET conceptualizes high-risk patients as the top 1% risk tier, which includes 10% of deaths by suicide.
REACH-VET, which uses a network of structured electronic health record (EHR) risk-variables ranging from health service usage to psychotropic medication usage, diagnoses, prior suicide prior attempts, and socio-demographics, is the VA's most sophisticated and far-reaching suicide prediction method (Jobes, Haddock, & Olivares, Reference Jobes, Haddock, Olivares, Ritchie and Llorente2019; Matarazzo, Brenner, & Reger, Reference Matarazzo, Brenner and Reger2019). That being said, REACH-VET's impact is constrained to the small subgroup of patients with known risk factors, like prior suicide attempts, opioid usage, and inpatient mental health treatment. Unfortunately, this leaves the majority of patients that go on to die by suicide without accurate risk classification or designated high-impact treatments (Mann, Michel, & Auerbach, Reference Mann, Michel and Auerbach2021; Stanley et al., Reference Stanley, Brown, Brent, Wells, Poling, Curry and Hughes2009). Recognizing this limitation, REACH-VET's authors acknowledge that their model inevitably fails to impact the majority of Veterans at risk for death by suicide whose risk falls outside of the high-risk tier:
To achieve substantial reductions in the burden of suicide, it will be necessary to target larger strata of patients at lower – but still elevated – risk; for example, the 5.00% of patients who account for 24% of suicides in VA patients over 1 year. Because so many patients fall into this stratum, and because of the magnitude of the resources that would be required for a comprehensive approach for these patients, demonstration projects and research are needed to develop and validate an array of risk-stratified interventions that can be realistically delivered across a health care system (McCarthy et al., Reference McCarthy, Bossarte, Katz, Thompson, Kemp, Hannemann and Schoenbaum2015).
Departing from the conventional focus on the high-risk population, this study analyzes VA patient suicide risk dimensions across risk tiers. REACH-VET, which uses a 0–100 percentile risk scoring system, with 100% being the lowest possible risk score, offers a potential method to identify not only the high-risk population, but also those with lower classified risk. In this study, we use these scores to evaluate classified risk, alongside demographic, social determinants of health, service usage, and diagnostic variables, to gain clinical and epidemiological understanding of all Veterans that died by suicide.
Following the REACH-VET authors' guidance, this study helps to lay the groundwork for future suicide prevention services for ‘larger strata of patients at lower – but still elevated – risk’. While lower-risk populations represent more than 90% of VA suicide decedents, to date they have been understudied and underserved. As an initial step toward reaching this community, this study uses national VA data to characterize patients across suicide risk strata to aid epidemiological understanding about risk dimensions and to help scaffold future development of evidenced-based suicide prevention services for all Veterans.
Methods
Sample selection
To develop the study sample, we linked EHR data from the VA Corporate Data Warehouse (CDW) EHR with cause of death data from the VA-Department of Defense Mortality Data Repository (MDR; VA/DoD, 2017) to identify all patients who died by suicide that had at least one VA health care encounter in either 2017 or 2018. REACH-VET scores are recalculated monthly for all VA patients. With support from the VA Office of Mental Health and Suicide Prevention, we identified each case's REACH-VET risk score during the month before death. Using CDW data, we pulled demographics, social determinants of health, service usage, prescriptions, and diagnostic information. In addition to REACH-VET, the VA uses a suicide risk warning system that can be manually designated by patients' clinicians (‘high-risk flag’) (Hein, Peltzman, Hallows, Theriot, & McCarthy, Reference Hein, Peltzman, Hallows, Theriot and McCarthy2021). We included high-risk flag indication from any time point within 3 months before death date as a descriptive variable within our analytic model to evaluate the overlap between algorithmic and clinical risk monitoring.
Analysis
There were three steps to our analysis. First, we evaluated suicide risk concentration as measured by REACH-VET. We identified the high-risk population as any patient with a ≤ 1 REACH-VET score based on the high concentration of suicide deaths within this tier (McCarthy et al., Reference McCarthy, Bossarte, Katz, Thompson, Kemp, Hannemann and Schoenbaum2015). Second, we used a data-driven method to assess patterns in demographics, diagnoses, and service usage among the remaining patients who died by suicide but were not identified as high-risk. We used principal component analysis (PCA; Jolliffe, Reference Jolliffe2002), an orthogonal linear transformation technique that maps data into coordinates based on projections of the greatest amounts of variance, to identify subgroups within the non-high suicide risk population. Variables with greater than 25% missing data were excluded from our PCA. Outliers were removed from the derived model based on standardized distribution (Serneels & Verdonck, Reference Serneels and Verdonck2008). Second, we investigate the link between patient characteristics and their REACH-VET scores in order to calculate patient risk tiers. We performed K-Means clustering on data transformed via PCA (Ding & He, Reference Ding and He2004). This enabled us to categorize patients into clusters that we hypothesized would correspond to varying levels of REACH-VET risk scores. To differentiate patients into moderate and low-risk groups effectively, we sought an optimal cut-off point within the REACH-VET percentile scores. This cut-off was derived by taking the mean of the lower quartile of REACH-VET scores in the first cluster and the upper quartile in the second cluster, an approach we have designated as the Q1–Q3 method. The subsequent risk tiers, delineated by this REACH-VET cut-off, demonstrated a high degree of association with the original clusters. This was corroborated using the adjusted Rand Index (Steinley, Reference Steinley2004), which accounts for chance when measuring the similarity of two clustering solutions: the one obtained through our unsupervised K-Means approach based on patient characteristics, and the other defined by our established REACH-VET cut-off. A comprehensive assessment of the adjusted Rand Index across various potential REACH-VET percentile cut-offs confirmed the appropriateness of the Q1–Q3 method in mirroring the initial clustering. Third, we compared our identified non-high-risk subgroups to the REACH-VET identified high-risk subgroup. We used odds ratios (OR) to compare subgroups using variables included within REACH-VET's predictive model, as well as additional demographics, social determinants of health, disability, mental health services, medical services, and diagnoses variables.
Results
We identified a total of 4810 cases. Within this sample, 9.4% (n = 452) were identified as ‘high-risk’ as indicated by REACH-VET risk percentile, as presented in Fig. 1. After excluding the high-risk subgroup, we analyzed the remaining non-high-risk population using PCA, which led to two components, as presented in Fig. 2. We established cut points on REACH-VET by averaging the first component's lowest quartile with the second component's highest quartile. This REACH-VET cut point allowed us to differentiate ‘moderate-risk’ (n = 2149 or 44.7% of sample), and ‘low-risk’ (n = 2209 or 45.9% of sample) subgroups, as presented in Fig. 3. Subgroup REACH-VET scores were significantly different (p < 0.001).
Figure 1. Plotting Veterans Affairs patient suicide deaths by Recovery Engagement and Coordination for Health – Veterans Enhanced Treatment (REACH-VET) (Cannizzaro, Reference Cannizzaro2017; Kessler et al., Reference Kessler, Hwang, Hoffmire, McCarthy, Petukhova, Rosellini and Bossarte2017) score and clinical high-risk indication (‘high-risk flag’) (Hein et al., Reference Hein, Peltzman, Hallows, Theriot and McCarthy2021) among patients that died by suicide in 2017 and 2018. The figure identifies the high-degree of overlap between REACH-VET, an algorothimic risk modeling technique, and high-risk flagging, a clinical risk modeling techinque.
Notes: RV is used in the figure title as an abbreviation for REACH-VET.
Figure 2. Principal components analysis (Jolliffe, Reference Jolliffe2002) of Veterans Affairs patient suicide deaths in 2017 and 2018, not-including those classified as high-risk by Recovery Engagement and Coordination for Health – Veterans Enhanced Treatment (REACH-VET; Cannizzaro, Reference Cannizzaro2017; Kessler et al., Reference Kessler, Hwang, Hoffmire, McCarthy, Petukhova, Rosellini and Bossarte2017).
Figure 3. Boxplot presenting principal components analysis (Jolliffe, Reference Jolliffe2002) derived clusters. Boxplots were used to identify the cut point between low-risk and moderate-risk patients' Recovery Engagement and Coordination for Health – Veterans Enhanced Treatment (REACH-VET; Cannizzaro, Reference Cannizzaro2017; Kessler et al., Reference Kessler, Hwang, Hoffmire, McCarthy, Petukhova, Rosellini and Bossarte2017) scores. The cut point was identified by averaging the first cluster's first REACH-VET quartile with the second cluster's third quartile. Clusters' REACH-VET scores were significantly different (p < 0.001).
When comparing the high, moderate, and low-risk subgroups, marked differences in demographics, social determinants of health, disability, mental health services, medical services, medications, and diagnoses were detected, as presented in Table 1. The high-risk subgroup included the plurality of patients who received clinical high-risk flags. Patients classified as high-risk tended to be younger, White, unmarried, experienced homelessness, and have elevated mental health burden, when compared to moderate- and low-risk patients. Patients that were classified as moderate- and low-risk tended to be older, married, Black, and Native American or Pacific Islander, and have elevated physical health burden. Lower-risk patients had substantially more missing data than high-risk patients.
Table 1. Veterans Affairs patients that died by suicide in 2017 and 2018 based on suicide risk subgroup
Subgroups were derived from Recovery Engagement and Coordination for Health – Veterans Enhanced Treatment (REACH-VET) (Cannizzaro, Reference Cannizzaro2017; Kessler et al., Reference Kessler, Hwang, Hoffmire, McCarthy, Petukhova, Rosellini and Bossarte2017) risk scores (high-risk = 1, moderate-risk = 2–24, and low-risk = 25–100). Analysis includes demographics, social determinants of health, disability, mental health services, medical services, and diagnoses variables.
Notes: Variables marked as ‘_prior12’ include data from the 12 months before death.
Variables marked as ‘_prior24’ include data from the 24 months prior to death.
When comparing high-risk and moderate-risk subgroups, we found that recent homelessness, elevated mental health burden, depression, substance abuse, and clinical high-risk flag substantially increased the odds of being in the high-risk subgroup. Conversely, we found that being married, over the age of 75, and having served during the Vietnam era substantially reduced the odds of being in the high-risk subgroup.
When comparing moderate-risk and low-risk subgroups, we found that recent homelessness, elevated mental health burden, conduct disorder, clinical high-risk flag, depression, and receiving an antidepressant prescription substantially increased the odds of being in the moderate-risk subgroup. Conversely, we found that being married, over the age of 75, being Black, Native American or Pacific Islander, having any missing variables, and having some disability substantially reduced the odds of being in the moderate-risk subgroup.
When comparing high-risk and low-risk subgroups, we found that elevated mental health, anxiety, depression, personality and conduct disorders, substance abuse, clinical high-risk flag, and receiving psychopharmaceutical treatment dramatically increased the odds of being in the high-risk subgroup. Conversely, we found that being Black, married, over the age of 75, and having some physical disability substantially reduced the odds of being in the high-risk subgroup. Full OR results are presented in Table 2.
Table 2. Odds ratios for risk subgroup classification (high-risk, moderate-risk, and low-risk) for Veterans Affairs patients that died by suicide in 2017 and 2018
Odd ratios with significant p values are marked in bold.
Notes: ap values calculated using Fisher's exact method.
Variables marked as ‘_prior12’ include data from the 12 months before death.
Variables marked as ‘_prior24’ include data from the 24 months prior to death.
Discussion
Given the challenge of predicting suicide risk, REACH-VET was designed to identify patients with the highest suicide risk concentration (Kessler, Bossarte, Luedtke, Zaslavsky, & Zubizarreta, Reference Kessler, Bossarte, Luedtke, Zaslavsky and Zubizarreta2020), a measurement defined by the REACH-VET authors as ‘the ratio of observed case patients to the case patients that would be expected if the distribution were uniform across strata’ (McCarthy et al., Reference McCarthy, Bossarte, Katz, Thompson, Kemp, Hannemann and Schoenbaum2015). Our high-risk subgroup confirms this distribution, accounting for a large swatch of patients in one risk stratum. As anticipated this ratio was much higher in the high-risk (9.4× greater than the risk rate in the overall sample) than the moderate-risk (1.9× greater than the risk rate in the overall sample) and low-risk (0.4× lower than the risk rate in the overall sample) subgroups. In addition to being identified as high-risk algorithmically, the high-risk subgroup tends to also be flagged by clinicians as being at high suicide risk nine times more than the moderate-risk and 58 times more than the low-risk subgroups. This high-risk subgroup tends to be younger, White, not married, and utilizes more mental health services and fewer medical services, when compared to patients with lower estimated risk. This relatively homogeneous subgroup stands in contrast to the much larger and more heterogeneous moderate-risk and low-risk subgroups, each of which is five times larger than the high-risk subgroup.
While there are obvious reasons to prioritize the high-risk subgroup's suicide prevention care, including their elevated risk concentration, mental health burden, relative demographic homogeneity, and prior utilization of VA mental health care, it is nonetheless imperative to prevent suicide across risk tiers, especially as the lower tiers represent the majority of suicide deaths. It is also necessary to recognize limitations associated with current prediction methods, shedding light on risk tier demographic disparities. To this end, we found that the high-risk subgroup included 10.06, 7.21, and 5.88% of the White, Native American or Pacific Islander, and Black patients. In contrast, the low-risk subgroup included 49.08, 63.10, and 64.3% of these respective populations. When looking at age distribution, we found that the high-risk subgroup included 13.11% of 18–34 and 1.8% of 75+ years-old patients, while the low-risk subgroup included 38.36 and 68.26% of these populations. These results indicate that select patient populations' risk burdens are undervalued within both algorithmic and clinical risk models, minimizing their ability to receive prevention care.
Representation
It is important to emphasize that REACH-VET's model was developed by identifying and weighting a network of machine learning-derived variables to best predict suicide for the subset of the VA patient population with the highest likelihood of dying by suicide (Kessler et al., Reference Kessler, Hwang, Hoffmire, McCarthy, Petukhova, Rosellini and Bossarte2017). Accordingly, REACH-VET offers less predictive relevance for some patients, especially those who come from demographic backgrounds that have lower suicide risk concentration or for patients who receive services that are not associated with suicide risk. Furthermore, REACH-VET works by leveraging patient EHR; accurate REACH-VET scoring therefore hinges on accurate EHR data. One of our consistent findings is that patients at lower-risk levels use fewer mental health services and have higher levels of missing data (Peltzman, Rice, Jones, Washington, & Shiner, Reference Peltzman, Rice, Jones, Washington and Shiner2022; Sullivan et al., Reference Sullivan, Simons, Mills, Hilgeman, Freytes, Morin and Byers2023). This could be associated with these patients accessing less services (Meffert et al., Reference Meffert, Morabito, Sawicki, Hausman, Southwick, Pietrzak and Heinz2019), preferring non-VA services (Mattocks et al., Reference Mattocks, Cunningham, Elwy, Finley, Greenstone, Mengeling, Pizer and Bastian2019), having greater barriers to care (Elnitsky et al., Reference Elnitsky, Andresen, Clark, McGarity, Hall and Kerns2013), or not disclosing personal information within care (Botero et al., Reference Botero, Rivera, Calloway, Ortiz, Edwards, Chae and Geraci2020). Providing these underrepresented patients, and subgroups of patients, with alternative suicide prediction and prevention mechanisms is a critical step in reducing patient suicide (Jobes et al., Reference Jobes, Haddock, Olivares, Ritchie and Llorente2019; Miller-Matero et al., Reference Miller-Matero, Yeh, Maffett, Mooney, Sala-Hamrick, Frank and Ahmedani2023).
The concern of over- and under-representation of select populations is a latent critique of machine learning and prediction methods (Huang, Galal, Etemadi, & Vaidyanathan, Reference Huang, Galal, Etemadi and Vaidyanathan2022). Like other machine learning classification metrics (Gianfrancesco, Tamang, Yazdany, & Schmajuk, Reference Gianfrancesco, Tamang, Yazdany and Schmajuk2018), REACH-VET may be constrained by concerns about lack of model transparency, biased training data, inconsistent analytic methods, and absence of clinical validations (Huang et al., Reference Huang, Galal, Etemadi and Vaidyanathan2022). As machine learning methods typically utilize pre-existing data, this approach frequently prioritizes historic trends over contemporary realities, replicating potential biases and service access limitations and skewing prediction models toward select race, age, and gender populations (Gianfrancesco et al., Reference Gianfrancesco, Tamang, Yazdany and Schmajuk2018; Huang et al., Reference Huang, Galal, Etemadi and Vaidyanathan2022; Nong, Williamson, Anthony, Platt, & Kardia, Reference Nong, Williamson, Anthony, Platt and Kardia2022). Indeed, disparities in mental healthcare utilization or underdiagnoses of depression and other suicide-related diagnoses among older adults and racial and ethnic minorities have been shown to contribute to misdiagnosis and under-hospitalization (Bailey, Mokonogho, & Kumar, Reference Bailey, Mokonogho and Kumar2019; Lavingia, Jones, & Asghar-Ali, Reference Lavingia, Jones and Asghar-Ali2020), factors that may weigh heavily in algorithmic risk modeling. Although a variety of analytic methods have been developed (Afrose, Song, Nemeroff, Lu, & Yao, Reference Afrose, Song, Nemeroff, Lu and Yao2022) that could alleviate some of modeling concerns, the efficacy of these approaches in regard to suicide prediction remains unknown. This study helps address these concerns by focusing on underrepresented patient populations with the intention of developing new prediction and prevention mechanisms that provide more equitable services.
Limitations
We specifically utilized a retrospective sample as this format was most sensible for studying VA suicide deaths. Given our focus on Veteran suicide, we intentionally only used Veteran data, and restricted non-Veteran generalization. Unfortunately, we have limited access to VA patients' usage of medical providers outside of the VA. Our models are somewhat constrained by the higher prevalence of missing data among low-risk patients than other risk subgroups.
Implications
Our study addresses select shortcomings associated with current high-risk suicide modeling methods and develops a framework to cluster lower-risk populations. Implications include expanding epidemiological understanding about Veteran suicide risk distribution. Our analysis highlighted the significant differences between risk subgroups, allowing identification of distinct care utilization trends and risk factors, information that could be subsequently used to scaffold tailored prevention programs. As current VA practice guidelines encourage utilizations of patient-centered and representative care models (VA/DoD, 2019), it is all the more important to prioritize the development of tailored prediction and prevention models for each of these subgroups. We are optimistic that this work can aid the development of adjunctive prediction models for populations that are underrepresented within the REACH-VET, including non-White, older patients, and non-mental health service users.
Given the VA's dedication to combating Veteran suicide, it is essential to reach as many patients as possible across all risk tiers. As a first step toward this goal, it will be important to gain clinical and epidemiological understanding about these patient populations and risk subgroups. Our prior work suggests the utility of leveraging additional EHR data formats, including unstructured EHR like provider notes, to improve clinical awareness and risk prediction accuracy (Levis et al., Reference Levis, Levy, Dent, Dufort, Gobbel, Watts and Shiner2023a; Levis, Levy, Dufort, Russ, & Shiner, Reference Levis, Levy, Dufort, Russ and Shiner2023b). This approach has been shown to offer specific benefits for patients that are missing data (Shiner et al., Reference Shiner, Levis, Dufort, Patterson, Watts, DuVall and Maguen2021) and could, accordingly, help mitigate limitations associated with non-high-risk patient data. Improved data, including unstructured EHR-derived variables, could aid risk clustering and expand epidemiological and clinical information. Future research could in turn utilize derived information to develop targeted prediction and prevention methods. Additionally, following World Health Organization guidance (Wasserman, Tadić, & Bec, Reference Wasserman, Tadić, Bec, Björnberg, Hansson, Belin and Tingvall2023), future research could utilize these materials to aid the development of universal suicide prevention strategies for low-risk patients (Klimes-Dougan, Klingbeil, & Meller, Reference Klimes-Dougan, Klingbeil and Meller2013) and selective suicide prevention strategies that aim to reach moderate-risk patients (Ahmedani & Vannoy, Reference Ahmedani and Vannoy2014), alongside indicated strategies for high-risk patients (McCarthy et al., Reference McCarthy, Bossarte, Katz, Thompson, Kemp, Hannemann and Schoenbaum2015). We are hopeful that identifying these trends can lead to more effective risk appraisal regardless of risk strata and, in turn, toward targeted suicide prevention care across all patient populations.
Data availability statement
Data access is restricted due to the clinical nature of dataset and VA privacy protections.
Funding statement
Dr Levis was supported by a VA New England Career Development Award (V1CDA-2020-60) and by a VA Clinical Science Research and Development Career Development Award (CX002630). Dr Levy was supported by a DoD award (HT9425-23-1-0267).
Competing interests
None.
Ethical standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
