Hostname: page-component-5cf477f64f-h6p2m Total loading time: 0 Render date: 2025-04-03T07:49:46.552Z Has data issue: false hasContentIssue false
  • English
  • Français

Natural language processing of clinical mental health notes may add predictive value to existing suicide risk models

Published online by Cambridge University Press:  17 February 2020

Maxwell Levis Open the ORCID record for Maxwell Levis [Opens in a new window] ,
Christine Leonard Westgate ,
Jiang Gui ,
Bradley V. Watts  and
Brian Shiner
Maxwell Levis*
Affiliation:
White River Junction VA Medical Center, White River Junction, VT, USA Geisel School of Medicine at Dartmouth, Hanover, NH, USA
Christine Leonard Westgate
Affiliation:
White River Junction VA Medical Center, White River Junction, VT, USA
Jiang Gui
Affiliation:
Geisel School of Medicine at Dartmouth, Hanover, NH, USA
Bradley V. Watts
Affiliation:
Geisel School of Medicine at Dartmouth, Hanover, NH, USA VA Office of Systems Redesign and Improvement, White River Junction, VT, USA
Brian Shiner
Affiliation:
White River Junction VA Medical Center, White River Junction, VT, USA Geisel School of Medicine at Dartmouth, Hanover, NH, USA VA Office of Systems Redesign and Improvement, White River Junction, VT, USA National Center for PTSD Executive Division, White River Junction, VT, USA
*
Author for correspondence: Maxwell Levis, E-mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Background

This study evaluated whether natural language processing (NLP) of psychotherapy note text provides additional accuracy over and above currently used suicide prediction models.

Methods

We used a cohort of Veterans Health Administration (VHA) users diagnosed with post-traumatic stress disorder (PTSD) between 2004–2013. Using a case-control design, cases (those that died by suicide during the year following diagnosis) were matched to controls (those that remained alive). After selecting conditional matches based on having shared mental health providers, we chose controls using a 5:1 nearest-neighbor propensity match based on the VHA's structured Electronic Medical Records (EMR)-based suicide prediction model. For cases, psychotherapist notes were collected from diagnosis until death. For controls, psychotherapist notes were collected from diagnosis until matched case's date of death. After ensuring similar numbers of notes, the final sample included 246 cases and 986 controls. Notes were analyzed using Sentiment Analysis and Cognition Engine, a Python-based NLP package. The output was evaluated using machine-learning algorithms. The area under the curve (AUC) was calculated to determine models' predictive accuracy.

Results

NLP derived variables offered small but significant predictive improvement (AUC = 0.58) for patients that had longer treatment duration. A small sample size limited predictive accuracy.

Conclusions

Study identifies a novel method for measuring suicide risk over time and potentially categorizing patient subgroups with distinct risk sensitivities. Findings suggest leveraging NLP derived variables from psychotherapy notes offers an additional predictive value over and above the VHA's state-of-the-art structured EMR-based suicide prediction model. Replication with a larger non-PTSD specific sample is required.

Keywords

Electronic medical recordsnatural language processingsuicide predictionveterans mental health
Type
Original Articles
Information
Psychological Medicine , Volume 51 , Issue 8 , June 2021 , pp. 1382 - 1391
Check for updates
Creative Commons
This is a work of the U.S. Government and is not subject to copyright protection in the United States.
Copyright
Copyright © The Author(s) 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Addis, ME. (2008). Gender and depression in men. Clinical Psychology: Science and Practice, 15(3), 153–168.Google Scholar
Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399–424.CrossRefGoogle ScholarPubMed
Barzilay, S., & Apter, A. (2014). Psychological models of suicide. Archives of Suicide Research, 18(4), 295–312.CrossRefGoogle ScholarPubMed
Beck, A., Steer, R., Kovacks, M., & Garrison, B. (1985). Hopelessness and eventual suicide: A 10-year prospective study of patients hospitalized with suicidal ideation. American Journal of Psychiatry, 1(42), 559–563.Google Scholar
Belsher, B. E., Smolenski, D. J., Pruitt, L. D., Bush, N. E., Beech, E. H., Workman, D. E., … Skopp, N. A. (2019). Prediction models for suicide attempts and deaths: A systematic review and simulation. JAMA Psychiatry, 76(6), 642–651.CrossRefGoogle ScholarPubMed
Ben-Ari, A., & Hammond, K. (2015). Text mining the EMR for modeling and predicting suicidal behavior among US veterans of the 1991 Persian Gulf War. 2015 48th Hawaii international conference on system sciences (pp. 3168–3175), January 2015.CrossRefGoogle Scholar
Bohnert, K. M., Ilgen, M. A., Louzon, S., McCarthy, J. F., & Katz, I. R. (2017). Substance use disorders and the risk of suicide mortality among men and women in the US Veterans Health Administration. Addiction, 112(7), 1193–1201.CrossRefGoogle ScholarPubMed
Brundin, L., Petersén, Å., Björkqvist, M., & Träskman-Bendz, L. (2007). Orexin and psychiatric symptoms in suicide attempters. Journal of Affective Disorders, 100(1–3), 259–263.CrossRefGoogle ScholarPubMed
Bulik, C. M., Carpenter, L. L., Kupfer, D. J., & Frank, E. (1990). Features associated with suicide attempts in recurrent major depression. Journal of Affective Disorders, 18(1), 29–37.CrossRefGoogle ScholarPubMed
Cambria, E., Havasi, C., & Hussain, A. (2012). SenticNet 2: A semantic and affective resource for opinion mining and sentiment analysis. Twenty-Fifth international FLAIRS conference, May 2012.Google Scholar
Cambria, E., Speer, R., Havasi, C., & Hussain, A. (2010). Senticnet: A publicly available semantic resource for opinion mining. 2010 AAAI fall symposium series, November 2010.Google Scholar
Cohen, R., Elhadad, M., & Elhadad, N. (2013). Redundancy in electronic health record corpora: Analysis, impact on text mining performance and mitigation strategies. BMC Bioinformatics, 14(1), 10–25.CrossRefGoogle ScholarPubMed
Colli, A., & Lingiardi, V. (2009). The Collaborative Interactions Scale: A new transcript-based method for the assessment of therapeutic alliance ruptures and resolutions in psychotherapy. Psychotherapy Research, 19(6), 718–734.CrossRefGoogle ScholarPubMed
Crossley, S. A., Kyle, K., & McNamara, D. S. (2017). Sentiment Analysis and Social Cognition Engine (SEANCE): An automatic tool for sentiment, social cognition, and social-order analysis. Behavior Research Methods, 49(3), 803–821.CrossRefGoogle ScholarPubMed
Dunster-Page, C., Haddock, G., Wainwright, L., & Berry, K. (2017). The relationship between therapeutic alliance and patient's suicidal thoughts, self-harming behaviours and suicide attempts: A systematic review. Journal of Affective Disorders, 223, 165–174.CrossRefGoogle ScholarPubMed
Elvins, R., & Green, J. (2008). The conceptualization and measurement of therapeutic alliance: An empirical review. Clinical Psychology Review, 28(7), 1167–1187.CrossRefGoogle Scholar
Fernandes, A. C., Dutta, R., Velupillai, S., Sanyal, J., Stewart, R., & Chandran, D. (2018). Identifying suicide ideation and suicidal attempts in a psychiatric clinical research database using natural language processing. Scientific Reports, 8(1), 7426.CrossRefGoogle Scholar
Forehand, J. A., Peltzman, T., Westgate, C. L., Riblet, N. B., Watts, B. V., & Shiner, B. (2019). Causes of excess mortality in veterans treated for posttraumatic stress disorder. American Journal of Preventive Medicine. 57(2), 145–152.CrossRefGoogle ScholarPubMed
Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1), 1–22.CrossRefGoogle ScholarPubMed
Ganzini, L., Denneson, L. M., Press, N., Bair, M. J., Helmer, D. A., Poat, J., & Dobscha, S. K. (2013). Trust is the basis for effective suicide risk screening and assessment in veterans. Journal of General Internal Medicine, 28(9), 1215–1221.CrossRefGoogle ScholarPubMed
Genuchi, M. C. (2019). The role of masculinity and depressive symptoms in predicting suicidal ideation in homeless men. Archives of Suicide Research, 23(2), 289–311.CrossRefGoogle ScholarPubMed
Geraci, J., Wilansky, P., de Luca, V., Roy, A., Kennedy, J. L., & Strauss, J. (2017). Applying deep neural networks to unstructured text notes in electronic medical records for phenotyping youth depression. Evidence-Based Mental Health, 20(3), 83–87.CrossRefGoogle ScholarPubMed
Hellmuth, J. C., Stappenbeck, C. A., Hoerster, K. D., & Jakupcak, M. (2012). Modeling PTSD symptom clusters, alcohol misuse, anger, and depression as they relate to aggression and suicidality in returning US veterans. Journal of Traumatic Stress, 25(5), 527–534.CrossRefGoogle Scholar
Heylighen, F. (1992). A cognitive-systemic reconstruction of Maslow's theory of self-actualization. Behavioral Science, 37(1), 39–58.CrossRefGoogle Scholar
Hom, M. A., Stanley, I. H., Podlogar, M. C., & Joiner, T. E. Jr (2017). ‘Are you having thoughts of suicide?’ Examining experiences with disclosing and denying suicidal ideation. Journal of Clinical Psychology, 73(10), 1382–1392.CrossRefGoogle ScholarPubMed
Hu, M, & Liu, B. (2004). Mining and summarizing customer reviews. In Kim, W, & Kohavi, R (Eds.), Proceedings of the tenth ACMSIGKDD international conference on knowledge discovery and data mining (pp. 168–177). Washington, DC: ACM Press.Google Scholar
Husky, M. M., Zablith, I., Fernandez, V. A., & Kovess-Masfety, V. (2016). Factors associated with suicidal ideation disclosure: Results from a large population-based study. Journal of Affective Disorders, 205, 36–43.CrossRefGoogle ScholarPubMed
Hutto, C. J., & Gilbert, E. (2014). Vader: A parsimonious rule-based model for sentiment analysis of social media text. Eighth international AAAI conference on weblogs and social media, May 2014.Google Scholar
Keller, S. M., Zoellner, L. A., & Feeny, N. C. (2010). Understanding factors associated with early therapeutic alliance in PTSD treatment: Adherence, childhood sexual abuse history, and social support. Journal of Consulting and Clinical Psychology, 78(6), 974–979.CrossRefGoogle ScholarPubMed
Kessler, R. C. (2019). Clinical epidemiological research on suicide-related behaviors – where we are and where we need to go. JAMA Psychiatry, 76(8), 777–778.CrossRefGoogle ScholarPubMed
Kessler, R. C., Bernecker, S. L., Bossarte, R. M., Luedtke, A. R., McCarthy, J. F., Nock, M. K., … Zuromski, K. L. (2019). The role of big data analytics in predicting suicide. In Passos, I., Mwangi, B., Kapczinski, F. (eds), Personalized psychiatry (pp. 77–98). Cham, CH: Springer Nature.CrossRefGoogle Scholar
Kessler, R. C., Hwang, I., Hoffmire, C. A., McCarthy, J. F., Petukhova, M. V., Rosellini, A. J., … Thompson, C. (2017). Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans Health Administration. International Journal of Methods in Psychiatric Research, 26(3), e1575.CrossRefGoogle ScholarPubMed
Kleiman, E. M., Turner, B. J., Fedor, S., Beale, E. E., Huffman, J. C., & Nock, M. K. (2017). Examination of real-time fluctuations in suicidal ideation and its risk factors: Results from two ecological momentary assessment studies. Journal of Abnormal Psychology, 126(6), 726–738.CrossRefGoogle ScholarPubMed
Koleck, T. A., Dreisbach, C., Bourne, P. E., & Bakken, S. (2019). Natural language processing of symptoms documented in free-text narratives of electronic health records: A systematic review. Journal of the American Medical Informatics Association, 26(4), 364–379.CrossRefGoogle ScholarPubMed
Lambert, M. J., & Barley, D. E. (2001). Research summary on the therapeutic relationship and psychotherapy outcome. Psychotherapy: Theory, Research, Practice, Training (new York, N Y), 38(4), 357–361.Google Scholar
Lasswell, H. D., & Namenwirth, J. Z. (1969). The Lasswell value dictionary. New Haven, CT: Yale Press.Google Scholar
Leonard Westgate, C., Shiner, B., Thompson, P., & Watts, B. V. (2015). Evaluation of veterans’ suicide risk with the use of linguistic detection methods. Psychiatric Services, 66(10), 1051–1056.CrossRefGoogle ScholarPubMed
Lowman, C. A. (2019). Optimizing clinical outcomes in VA mental health care. In Ritchie, E., Llorente, M. (eds), Veteran psychiatry in the US (pp. 29–48). New York, NY: Springer.CrossRefGoogle Scholar
Mallinckrodt, B., & Tekie, Y. T. (2016). Item response theory analysis of Working Alliance Inventory, revised response format, and new Brief Alliance Inventory. Psychotherapy Research, 26(6), 694–718.CrossRefGoogle ScholarPubMed
Martinez, V. R., Flemotomos, N., Ardulov, V., Somandepalli, K., Goldberg, S. B., Imel, Z. E., … Narayanan, S. (2019). Identifying therapist and client personae for therapeutic alliance estimation. Proc. Interspeech (pp. 1901–1905).CrossRefGoogle Scholar
Maslow, A. H. (1943). A theory of human motivation. Psychological Review, 50(4), 370–396.CrossRefGoogle Scholar
McCarthy, J. F., Bossarte, R. M., Katz, I. R., Thompson, C., Kemp, J., Hannemann, C. M., … Schoenbaum, M. (2015). Predictive modeling and concentration of the risk of suicide: Implications for preventive interventions in the US Department of Veterans Affairs. American Journal of Public Health, 105(9), 1935–1942.CrossRefGoogle ScholarPubMed
McGuire, J., & Rosenheck, R. (2005). The quality of preventive medical care for homeless veterans with mental illness. Journal for Healthcare Quality, 27, 26–32.CrossRefGoogle ScholarPubMed
McKinney, J. M., Hirsch, J. K., & Britton, P. C. (2017). PTSD Symptoms and suicide risk in veterans: Serial indirect effects via depression and anger. Journal of Affective Disorders, 214, 100–107.CrossRefGoogle ScholarPubMed
Mohammad, S. M., & Turney, P. D. (2010). Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. Proceedings of the NAACL HLT 2010 workshop on computational approaches to analysis and generation of emotion in text (pp. 26–34). Association for Computational Linguistics, June 2010.Google Scholar
Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3), 436–465.CrossRefGoogle Scholar
Narain, K., Bean-Mayberry, B., Washington, D. L., Canelo, I. A., Darling, J. E., & Yano, E. M. (2018). Access to care and health outcomes among women veterans using veterans administration health care: Association with food insufficiency. Women's Health Issues, 28(3), 267–272.CrossRefGoogle ScholarPubMed
Norcross, J. C., & Lambert, M. J. (2018). Psychotherapy relationships that work III. Psychotherapy, 55(4), 303–315.CrossRefGoogle ScholarPubMed
Pennebaker, J. W., Booth, R. J., & Francis, M. E. (2007). Linguistic inquiry and word count: LIWC [Computer software]. Austin, TX: liwc.net, 135.Google Scholar
Poulin, C., Shiner, B., Thompson, P., Vepstas, L., Young-Xu, Y., Goertzel, B., … McAllister, T. (2014). Predicting the risk of suicide by analyzing the text of clinical notes. PLoS One, 9(1), e85733.CrossRefGoogle ScholarPubMed
Qin, P., & Nordentoft, M. (2005). Suicide risk in relation to psychiatric hospitalization: Evidence based on longitudinal registers. Archives of General Psychiatry, 62(4), 427–432.CrossRefGoogle ScholarPubMed
Rogers, J. R. (2001). Theoretical grounding: The ‘missing link’ in suicide research. Journal of Counseling & Development, 79(1), 16–25.CrossRefGoogle Scholar
Rudd, M. D., Berman, A. L., Joiner, T. E. Jr, Nock, M. K., Silverman, M. M., Mandrusiak, M., … Witte, T. (2006). Warning signs for suicide: Theory, research, and clinical applications. Suicide and Life-Threatening Behavior, 36(3), 255–262.CrossRefGoogle ScholarPubMed
Rumshisky, A., Ghassemi, M., Naumann, T., Szolovits, P., Castro, V. M., McCoy, T. H., & Perlis, R. H. (2016). Predicting early psychiatric readmission with natural language processing of narrative discharge summaries. Translational Psychiatry, 6(10), e921.CrossRefGoogle ScholarPubMed
Scherer, K. R. (2005). What are emotions? And how can they be measured? Social Science Information, 44(4), 695–729.CrossRefGoogle Scholar
Schinka, J. A., Schinka, K. C., Casey, R. J., Kasprow, W., & Bossarte, R. M. (2012). Suicidal behavior in a national sample of older homeless veterans. American Journal of Public Health, 102(Suppl 1), S147–S153.CrossRefGoogle Scholar
Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464.CrossRefGoogle Scholar
Shiner, B., Leonard Westgate, C., Bernardy, N. C., Schnurr, P. P., & Watts, B. V. (2017). Trends in opioid use disorder diagnoses and medication treatment among veterans with posttraumatic stress disorder. Journal of Dual Diagnosis, 13(3), 201–212.CrossRefGoogle ScholarPubMed
Shiner, B., Westgate, C. L., Gui, J., Cornelius, S., Maguen, S. E., Watts, B. V., & Schnurr, P. P. (2019). Measurement strategies for evidence-based psychotherapy for posttraumatic stress disorder delivery: Trends and associations with patient-reported outcomes. Administration and Policy in Mental Health and Mental Health Services Research, 1–17. https://doi.org/10.1007/s10488-019-01004-2.Google Scholar
Stone, P. J., Dunphy, D. C., Smith, M. S., & Ogilvie, D. M. (1966). The general inquirer: A computer approach to content analysis: Studies in psychology, sociology, anthropology, and political science. Cambridge, MA: MIT Press.Google Scholar
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267–288.Google Scholar
Torous, J., Larsen, M. E., Depp, C., Cosco, T. D., Barnett, I., Nock, M. K., & Firth, J. (2018). Smartphones, sensors, and machine learning to advance real-time prediction and interventions for suicide prevention: A review of current progress and next steps. Current Psychiatry Reports, 20(7), 51–67.CrossRefGoogle ScholarPubMed
Urbanowicz, R. J., & Moore, J. H. (2009). Learning classifier systems: A complete introduction, review, and roadmap. Journal of Artificial Evolution and Applications, 2009(1), 1–25.CrossRefGoogle Scholar
Van Orden, K. A., Witte, T. K., Cukrowicz, K. C., Braithwaite, S. R., Selby, E. A., & Joiner, T. E. Jr (2010). The interpersonal theory of suicide. Psychological Review, 117(2), 575–600.CrossRefGoogle ScholarPubMed
VA Office of Public and Intergovernmental Affairs. (2017). VA Office of Public and Intergovernmental Affairs. VA REACH VET Initiative helps save veterans lives: Program signals when more help is needed for at-risk veterans. Retrieved from https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/3527-notes.pdf/.Google Scholar
Verona, E., Patrick, C. J., & Joiner, T. E. (2001). Psychopathy, antisocial personality, and suicide risk. Journal of Abnormal Psychology, 110(3), 462–470.CrossRefGoogle ScholarPubMed
Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2017). Predicting risk of suicide attempts over time through machine learning. Clinical Psychological Science, 5(3), 457–469.CrossRefGoogle Scholar
Wang, E. A., McGinnis, K. A., Goulet, J., Bryant, K., Gibert, C., Leaf, D. A., … Fiellin, D. A. (2015). Food insecurity and health: Data from the Veterans Aging Cohort Study. Public Health Reports, 130(3), 261–268.CrossRefGoogle ScholarPubMed
Widome, R., Jensen, A., Bangerter, A., & Fu, S. S. (2015). Food insecurity among veterans of the US wars in Iraq and Afghanistan. Public Health Nutrition, 18(5), 844–849.CrossRefGoogle ScholarPubMed
Wilks, C. R., Morland, L. A., Dillon, K. H., Mackintosh, M. A., Blakey, S. M., Wagner, H. R., … Elbogen, E. B. (2019). Anger, social support, and suicide risk in US military veterans. Journal of Psychiatric Research, 109, 139–144.CrossRefGoogle Scholar
World Health Organization. (2018). Retrieved from http://www.who.int/mental_health/prevention/suicide/sui-cideprevent/en/.Google Scholar