Monitoring Mental Health: Legal and Ethical Considerations of Using Artificial Intelligence in Psychiatric Wards

Abstract

Artificial intelligence (AI) is being tested and deployed in major hospitals to monitor patients, leading to improved health outcomes, lower costs, and time savings. This uptake is in its infancy, with new applications being considered. In this Article, the challenges of deploying AI in mental health wards are examined by reference to AI surveillance systems, suicide prediction and hospital administration. The examination highlights risks surrounding patient privacy, informed consent, and data considerations. Overall, these risks indicate that AI should only be used in a psychiatric ward after careful deliberation, caution, and ongoing reappraisal.

1. Introduction

Global health care systems are under overwhelming pressure to meet the exponentially increasing demand for their services. Mental disorders significantly contribute to such increases in demand in the mental health care context. ¹ A crucial element of mental health care is acute inpatient care. In order to provide quality and safe care, effective monitoring and management of inpatients in mental health or psychiatric wards is essential. ² Since patient monitoring is typically a labor-intensive activity, inpatient psychiatric units have faced distinctive challenges in this area, ranging from slow and inaccurate decision-making to inadequate availability of the right information at the right time. ³ The growing demand for mental health care coupled with limited monitoring resources has created opportunities for promising digital solutions such as artificial intelligence (AI) to help address those challenges. The technology extracts novel insights from vast accumulated data about individuals (i.e., big data) to make predictions and recommendations. ⁴

This Article explores the practical, legal, and ethical challenges of deploying AI to provide care and operational efficiency in psychiatric units. ⁵ While AI systems have not yet gained specific approval for use in mental health wards, research on its use is presently being undertaken in the United States, the United Kingdom, and other countries. ⁶ As such, it is a timely opportunity to flag the relevant legal risks distinct to the psychiatric setting before such systems are used in practice.

This exploration is undertaken in three parts. First, there is an examination of the use of AI in hospital wards generally and the associated legal concerns. Second, the distinction between those concerns in psychiatric wards and other wards is outlined. While there is some crossover between the issues, there are distinct factors concerning privacy, informed consent, and data considerations. ⁷ Three scenarios are examined to elucidate the challenges in the mental health context: namely, the use of AI-CCTV camera systems to monitor patients, the use of suicide prediction tools, and the use of AI for administration in mental health wards. These scenarios are intended to elucidate the crucial legal considerations; they are not intended to be exhaustive of all potential matters that may arise in a psychiatric ward. Third, the Article recommends developing guidelines that account for the highlighted risks should AI systems eventually be used in psychiatric wards. Overall, significant caution should be taken in implementing AI in psychiatric wards due to the risks posed to patients. While specific use cases may benefit patients, AI should be used only in narrow contexts following robust legal processes.

2. The Use of AI in Hospitals

A critical inquiry is the extent to which there is a distinction between the legal and ethical concerns of using AI in psychiatric wards and other hospital wards. This inquiry is important because mental health is often treated separately from physical health. Many countries have dedicated mental health policies and laws, such as the Mental Health Act 1983 in England and Wales, with provisions on hospital admission. The question is whether such distinctions may result in different requirements for AI used for inpatients in psychiatric wards compared to other wards. The analysis below indicates that there is overlap, but some pertinent additional considerations arise surrounding patient privacy, informed consent and data. To understand those considerations, it is first helpful to consider some existing applications of AI systems in hospitals.

In hospitals, camera-based systems used to track patients’ movements now feature built-in AI algorithms designed to “instantly recognize any improper movement based on pre-defined data templates.” ⁸ Cameras with intelligent sensors can be trained using historical data to spot specific scenarios, ⁹ and predict patient behavior, forecast the number of people expected to visit an ER or a waiting room, identify the most frequently used areas of the hospitals and manage patient and visitor traffic around the hospital to improve patient flow. ¹⁰

Such AI systems are being deployed in hospitals around the world. For example, researchers at Eindhoven University of Technology developed an intelligent camera that monitors patients around-the-clock and can detect subtle changes in a patient’s face or chest. The AI-enabled camera notifies doctors and nurses of possible complications, allowing timely intervention, which can reduce unexpected deaths in the hospital. ¹¹ A similar application was used in a hospital in Nebraska where AI-enabled cameras were installed across beds for patients deemed at risk of leaving their beds unattended and falling. Around 200,000 hours of video recording data were used to train the AI algorithm to predict when a patient is about to leave the bed and alert the staff on duty if intervention is required. ¹²

AI systems are also enhancing operational efficiency and improving security around hospitals. Cameras with facial recognition technology can “identify and authenticate individuals to control access and prevent abduction.” ¹³ Great Ormond Street Hospital (GOSH) in the United Kingdom partnered with technology company Arm to develop smart cameras that not only predict a person’s behavior but can also detect intruders by cross-checking a given individual’s physical features with a database of known faces and alert security. The AI system is also expected to assist doctors during surgeries in the future by monitoring equipment usage and ensuring surgical tools are not accidentally left inside patients. The system does not store any live video information or pictures, thus protecting patients’ identities. ¹⁴ AI personal assistants may also aid patient monitoring when medical staff are unavailable. ¹⁵ Indeed, AI digital assistant monitors have been used at the Cleveland Clinic to monitor 100 beds in 6 ICUs. These systems identify at-risk patients through predictive and advanced analytics by monitoring and learning from conversations between doctors and patients in hospital rooms. ¹⁶

Furthermore, AI can predict workloads in emergency departments ¹⁷ and match competencies of health care providers to specific tasks for efficient staff rota planning. ¹⁸ Thus, AI systems capture video data from physical spaces where sensitive patient care activities take place and interpret the behaviors of health care professionals and patients, to monitor deviations in intended bedside practices, patient mobilisation activities and hygiene, ¹⁹ as well as to conduct bed occupancy checks, patients’ vitals monitoring, and patient fall or violence detection, among others. ²⁰

2.1 The Legal Concerns of Using AI in Hospitals

While continuous AI monitoring reduces the need for direct patient observation and frees hospital staff from administrative tasks, it is not without challenges. ²¹ Gerke et al. have mapped out the ethical and legal concerns that broadly arise in AI and health care. ²² Ethical concerns include informed consent to use, safety and transparency, algorithmic fairness and bias, and data privacy. Legal concerns include safety and effectiveness, liability, data protection and privacy, cybersecurity, and intellectual property. ²³ Gerke et al. narrow those considerations when examining aspects of ambient intelligence in hospitals to privacy and reidentification risks, consent and liability. ²⁴ To that, we can add data accuracy concerns.

On privacy and reidentification, the risk is that health care professionals and patients may be identified, which could undermine their privacy. While captured data may be deidentified in one data set (for example, by capturing silhouette images instead of color images), data triangulation makes it possible to reidentify those individuals from other data sets (such as doctor shift rota data). ²⁵ Additionally, data accuracy risks are associated with smart surveillance due to insufficient data available to train AI algorithms. Developing these algorithms is a cognitively laborious task, which requires obtaining sufficient training examples of scant activities undertaken by the staff and patients around the hospital and manually annotating data from video recordings, which is susceptible to human labelling error. ²⁶ Data bias is also a potential pitfall, with some studies finding that algorithms demonstrate significant racial bias when predicting aspects of care. ²⁷

Data accuracy is also affected by practical considerations, such as the camera placement in a hospital ward. The location of smart cameras determines an AI-enabled system’s effectiveness in capturing and processing accurate real-time data. ²⁸ In this regard, two main concerns arise. Firstly, the architectural design of different hospital units does not allow the hospital to install a network of sensors with sufficient overlapping fields of view, making data association and trajectory prediction across all hospital units a cumbersome task. Secondly, limited visual data affects accuracy because important visual appearance cues are lost as de-identified depth images are used instead of color videos due to privacy concerns surrounding personal health information, making tracking personnel extremely difficult. ²⁹

These accuracy risks could feed into medical liability concerns. For example, AI systems could make incorrect recommendations that may ultimately harm patients, raising questions about where responsibility lies. ³⁰ Patients may also require access to data recorded from an AI system for a liability claim, which raises questions about the appropriate rules a hospital implements for retaining and deleting data. Hospital workers could fail to act on a problem detected by an AI system, which could leave the hospital vulnerable to liability claims. AI systems may also record illegal activities, which requires clear policies about the circumstances in which hospitals will retain and share the recordings with law enforcement. ³¹

On consent, the concern is whether explicit consent should be required from patients for AI systems recording them. ³² Hospitals have been sued for recording and retaining videos of patients in vulnerable situations in U.S. hospitals. ³³ Nevertheless, hospitals might escape liability using consent documents containing boilerplate language, which provide authorization for hospitals to record patients for quality improvement and security reasons. ³⁴ It has been noted that the more explicit and specific the consent is, the better. ³⁵ For hospital employees, the employer has a “strong claim of a right to record their employee without meaningful consent” owing to legitimate business interest and public interest justifications under U.S. law. ³⁶ That right to record the employee will be satisfied as long as the recording is taken in a quasi-public space such as an operating room, corridor or patient room, where the goal is to improve patient care, where workers are informed about the recording, and where proper safeguards are in place to protect individual privacy. ³⁷ As such, some literature has examined the legal concerns arising in the context of AI being used in hospitals. The query here is the extent to which there is a distinction in the mental health inpatient context.

3. The ‘Siloed’ Considerations in Mental Health

Examining AI systems specific to the inpatient mental health context is challenging. Neither the FDA nor any other regulatory body has approved such devices for use in psychiatric wards (as of 2023). ³⁸ One useful starting point is to consider the rules that apply to research on mental health because significant research is being undertaken involving psychiatric inpatients with a view to deploying the technology in the future. Taking a step back, we can map out important international norms that bind such research. When developing AI systems, developers will train those systems on data. Researchers can obtain real-time data from individuals on-site at a hospital or off-site in naturalistic settings such as the home. ³⁹ User consent for data collection is required, as are the expected Institutional Review Board (IRB) approvals. ⁴⁰

Mental health data is often regarded as more sensitive than other health data; it is, therefore, covered by the Helsinki Declaration on protecting vulnerable patient groups, such as psychiatric patients. ⁴¹ Medical research with a vulnerable group is only justified where it is responsive to the health needs of that group, and “should stand to benefit” the group. ⁴² The phrasing of “should stand to benefit” amends the previous threshold that such research is justified where there is a “reasonable likelihood” that the population stands to benefit. ⁴³ This change broadened the justificatory scope for research involving vulnerable populations such as psychiatric patients. Informed consent is critical in this context. Consent must be voluntary or, if not possible, given by a legally authorized representative. ⁴⁴ Additionally, there are well-established principles of the United Nations on protecting persons with mental illness and improving mental care that are relevant to research. ⁴⁵ Every patient in a mental health facility shall have the right to privacy. ⁴⁶ Clinical trials and experimental treatments shall never be carried out on any patient without informed consent unless approval is given from a competent, independent review body specifically constituted for that purpose. ⁴⁷

Thus, from a research perspective, mental health patients are treated with specific norms and considerations in mind. There is also a legal distinction between the rules covering psychiatric patients and other patients. In mental health, specific rules govern diagnostic considerations, voluntary and involuntary admission and treatment, informed consent for special treatments such as electroconvulsive therapy, monitoring and review mechanisms, and the governance and administration of health care services, among others. ⁴⁸ Other international norms overarch laws on detention, such as Article 8(1) of the European Convention on Human Rights (ECHR) on the right to respect for private, family life, home and correspondence, which creates an obligation on staff in psychiatric wards to balance the right to privacy of patients with safety. ⁴⁹ There is also Article 3 on the right to be free from torture and inhuman and degrading treatment and Article 14 on the right to be free from discrimination.

The “siloed” legal approach in mental health has been criticized for emphasizing segregation, potentially reinforcing stigma, and creating exceptions to the equal exercise of patient rights through arbitrary restrictions and the right to free and informed consent. ⁵⁰ Nevertheless, the legal distinction exists. Further, while informed consent is required for any patient in a hospital partaking in such research, the law and guidelines deem psychiatric patients more vulnerable, requiring more checks and balances. The query is how these general principles governing mental health research, and the specific legal provisions in the mental health space, intersect with the legal considerations for AI systems. Below, these considerations are elucidated through an examination of three applications of AI systems in the psychiatric inpatient setting: namely, AI surveillance, suicide prediction, and hospital administration. These applications are offered as illustrative case studies for the analyses in this Article and are not intended to be comprehensive of all potential applications.

3.1 AI-CCTV Surveillance in Hospitals

The manner in which surveillance is implemented is very important to the patient experience. CCTV cameras are a crucial tool for monitoring communal areas shared by patients and staff and seclusion rooms. CCTV may, in certain circumstances, be used in private areas, including patient bedrooms. ⁵¹ Continuous CCTV monitoring enables better risk detection in hospital wards, allows patient observations to occur without disturbance, and provides data for training purposes. ⁵² However, surveillance “within hospital wards is not based on reciprocity” as it subjects patients to constant monitoring without their consent. ⁵³ Hospitals have adopted numerous policies and guidelines generally addressing surveillance, but additional, more specific considerations on privacy and informed consent apply in psychiatric wards, suggesting that much caution should be exerted when monitoring psychiatric patients with AI. ⁵⁴ This phenomenon is easily observed when examining the rubric of rules that have developed in England. There, the Home Office has developed a Surveillance Camera Code of Practice that emphasizes CCTV should only be used in the pursuit of a legitimate aim, be necessary, take account of an individual’s privacy and be subject to regular review, among other rules. ⁵⁵ In the health sphere, the Care Quality Commission (CQC) and Information Commissioner’s Office (ICO) provide guidance and regulatory oversight for care in England. The ICO can investigate and take enforcement action where surveillance breaches a patient’s rights under the Data Protection Act (DPA) 2018 and U.K. GDPR. To assist organizations, it has developed detailed guidance on the use of CCTV and surveillance. ⁵⁶

Privacy and dignity must always be maintained with surveillance. ⁵⁷ Cameras can be installed in public and communal areas, but not private areas such as bathrooms. However, covert recordings are permissible if the benefits outweigh other considerations on privacy and dignity. ⁵⁸ All recordings must be logged and traceable and not retained for more than 28 days. ⁵⁹ Disclosure of recordings can only be made in strict adherence to the DPA 2018 and U.K. GDPR. ⁶⁰ Where an individual cannot consent to the use of CCTV, the clinical leads must decide according to the Mental Capacity Act (MCA) 2005. ⁶¹ Thus, if CCTV is to be used in a patient’s bedroom (especially where a patient lacks mental capacity), permission may be required from the Court of Protection on an ad-hoc basis. ⁶²

Also of import is the ECHR. Article 8 articulates a qualified right, which means that recordings can be used despite breaching privacy where the recording is lawful, for a legitimate aim, and proportionate. ⁶³ The least restrictive possible alternatives to CCTV must be considered, such as using call bells or pendants, using sensors that detect movements, improving staff training and supervision, or increasing staff numbers. ⁶⁴ Depending on the system used, these considerations may be a double-edged sword for AI. CCTV incorporating AI technology may be less likely to gain approval, but an AI-enabled wearable may be more likely to gain approval because it may be deemed less restrictive.

The British Institute of Human Rights (BIHR) provide case studies for these considerations. They discuss a man with autism who sometimes harmed himself. Staff installed a CCTV camera in his room because that was deemed less restrictive than posting a staff member in his room twenty-four hours a day; they determined that would be a legitimate and proportionate response to the circumstances. The staff then installed CCTV in all the rooms of the facility, but they were not switched on. The courts considered the matter and decided that CCTV could be used in the man’s bedroom, but its use was deemed a significant restriction of his Article 8 ECHR rights, requiring that a judge regularly review the use of the CCTV camera in his bedroom. It was held that using CCTV in other bedrooms was not lawful, legitimate, or proportionate, so they were removed. ⁶⁵

Such examples raise questions about the appropriate limits of surveillance. It is wrong to assume that psychiatric patients require higher levels of supervision. Applying blanket surveillance rules without considering the impact on individuals could cause more harm and distress than help. ⁶⁶ This was the case for another patient detained under the law and subject to CCTV surveillance installed in her room and every other room in addition to “arm’s length” observations by two staff day and night. The patient found the CCTV to be intrusive, inhumane and degrading. ⁶⁷ Consequently, surveillance must be carefully planned, considering what is legitimate, necessary, and proportionate. A major trade union has recommended to members that privacy impact assessments be conducted. ⁶⁸ The CQC has noted that it is up to the providers to decide whether to use surveillance (within the limits of the law), but that “we would be concerned by an over-reliance on surveillance to deliver key elements of care, and it can never be a substitute for trained and well supported staff.” ⁶⁹

While such legal protections exist in England, their implementation can be problematic; privacy can be quite limited. The Mental Health Act Revised Code of Practice places an obligation on public authorities to respect a person’s right to a private life, including those detained under the Act. ⁷⁰ However, CCTV monitoring of communal areas used by patients—not all of whom have consented to such surveillance—raises privacy concerns. The right to privacy is further curtailed for involuntarily detained psychiatric patients. Since their access to space outside the ward environment is restricted due to the involuntary nature of admission, such patients have limited areas where they can avoid constant monitoring. ⁷¹ These risks may be replicated when AI is incorporated into such technologies. The risk of constant surveillance remains and may be heightened if the form of monitoring encompasses wearable devices that constantly track a patient, no matter their location in a ward.

Despite these concerns, the research in this area is not clear-cut. One study in Australia notes that CCTV cameras do not necessarily upset patients or motivate them to spend time in locations without camera surveillance. ⁷² Some patients may be passive about CCTV monitoring, which could make them feel safe, particularly on open wards. ⁷³ The views of staff should also be considered. Many staff in psychiatric wards have been victims of assault by patients, and surveillance could protect them. ⁷⁴ Staff are also concerned about how to use AI systems to identify such risks. ⁷⁵ The key is to implement security measures that do not provoke aggression or violence. In this regard, the characteristics of the ward are essential, and due consideration should be given to how AI systems could and should fit within that environment. The CQC has noted that many psychiatric units are in old and outdated buildings, which can lead to issues around privacy and dignity for patients. Wards can be “barren, visually impoverished environments dominated by security fencing” that can undermine morale and have a detrimental impact on recovery. ⁷⁶ The same is true of most psychiatric wards in the world, which are in old buildings that do not allow for proper CCTV surveillance. The use of AI tracking systems in those environments should, therefore, only be done with careful planning.

Another challenge is gaining informed consent for using CCTV cameras due to psychiatric patients’ varying levels of perception and cognition which can result in an impaired or fluctuating capacity to consent. Certain conditions, such as those linked to feelings of paranoia, tend to incite violent reactions towards the existence of cameras. ⁷⁷ While it is good practice under the MCA to obtain informed consent for CCTV monitoring, how and when such consent should be sought is ambiguous. Although there is a general presumption of surveillance in public spaces at hospitals not requiring consent, the same cannot be said when CCTV is used to monitor communal areas, seclusion rooms or patient bedrooms. These risks are deepened and broadened where AI is used because of the more pervasive and detailed information that such systems may potentially generate. Even if there is consent for using an AI system in a specific way, the autonomous nature of the system could result in recommendations and analyses for which consent was not obtained initially.

Thus, it is challenging to determine where the balance should tilt on using AI-enabled surveillance systems. There are obvious risks and strict legal considerations for the use of CCTV generally, but should those risks entirely preclude their use for psychiatric inpatients? AI-enabled surveillance does not yet exist in mental health wards, but one could consider prison surveillance by analogy. AI surveillance is used on prisoners to detect and prevent harmful incidences in China, Hong Kong, the United Kingdom, the United States, India and South Korea. ⁷⁸ Some systems can detect changes in parameters up to thirty minutes before an observable act of aggression with greater precision than existing methods. ⁷⁹ In Hong Kong, AI surveillance systems can detect instances of self-harm and alert staff, and detect if a person has any contraband. ⁸⁰ Wearable wristbands can track vital signs and flag any signs of injury. Robots in South Korea can monitor violence and suicide risks. ⁸¹ In the United States, mass-monitoring millions of phone calls through AI has helped officials combat violence, drug smuggling and attempted suicides in near real-time. By searching for keywords, phrases, and prison slang, investigators notify law enforcement officers when the AI system picks up suspicious language. This has prevented dozens of attempted suicides by providing inmates with psychological counselling in the minutes and hours after the inmate was recorded making references to harming themselves, threatening violence, or plotting to smuggle in contraband. ⁸²

Of course, the analogy cannot be stretched too far. Inpatients are not inmates in prisons, and one should not generalize the characteristics of psychiatric inpatients to be at risk of suicide or harm to others. Some inpatients are voluntarily there. Nevertheless, there are some similar paradigms and risks that exist that are helpful to consider, such as patients being involuntarily detained, those at risk of self-harm or suicide, those possessing contraband that can be a risk to themselves or others, and those posing a risk of violence (a common occurrence on psychiatric wards). ⁸³ AI may be helpful in those instances, and there is value in further research being conducted in this area. From a legal standpoint, applying England’s laws as an example, the conclusion that can be drawn at present is that, where a dispute or question arises for a court to consider, a surveillance system in mental health wards may require approval from a court and may be subject to constant review by a judge. However, from a practical standpoint, those legal protections may not always manifest, meaning privacy concerns will persist. In most cases, it appears that surveillance is used without questions arising for a court to consider, and there is a risk that AI-enabled systems will fall within the same void despite the greater legal questions or risks those systems may pose owing to their pervasive nature.

3.2 Predictive Behavior Models for Suicide

Many patients are detained under mental health laws for treatment, and many of them pose a considerable risk to themselves or others. ⁸⁴ Given the frequency of violence and suicide attempts in psychiatric units, accurate prediction of such behaviors is critical for proper treatment. ⁸⁵ The current structure for treatment relies on predictive models that are rarely specific to an individual, instead “classifying a patient to a certain group and referring to that group’s averages.” ⁸⁶ Traditional statistical models accommodate a limited number of variables, limiting predictions based on surrogate markers of suicide attempts, suicidal planning and ideation, and self-injury. ⁸⁷ Psychiatric units also use risk assessment instruments to predict aggressive behavior. ⁸⁸ These instruments use individual medical and social history data, treatment conditions, behaviors, and psychopathology. ⁸⁹ Yet, these tools are “based on one-off assessments of risk factors, use electronic or paper forms, and only predict risk over the very short-term (i.e. the next 24 hours)”. ⁹⁰ Moreover, these factors become clinically irrelevant in an emergency where the medical history of a possibly violent detained patient is unknown. ⁹¹ Other common methods include analyzing EHRs, national registries and self-reported questionnaires, although the latter’s reliability is questionable due to the patient’s motivation, capacity, and self-awareness. ⁹²

Overall, the predictive ability of suicide has not improved in fifty years of research. ⁹³ Suicide prediction remains incorrect in most cases. ⁹⁴ Yet eighty-three percent of people who die by suicide have contact with health services the year before their death, and forty-five percent have contact the month before. ⁹⁵ Therefore, there is a significant opportunity to assist doctors in assessing the risk of suicide using AI when those patients present. ⁹⁶

AI could be used in hospitals as a decision support for assessing suicide risk using surveillance or by examining EHRs, hospital records, and other government data sources. ⁹⁷ Inpatients may also provide data using smartphones to log their mood, sleep patterns, step count, and interactions with other patients. ⁹⁸ At the same time, using such systems could pose risks for patients in two realms: first, in research, when AI systems are being developed; second, in the ward, once AI has been deployed.

On research in the mental health context generally (not only inpatients), Marks has examined AI-based suicide prediction algorithms and the risks they pose. ⁹⁹ Marks highlights AI tools that examine medical records to predict suicide and social interactions that fall outside the health care sphere. ¹⁰⁰ Focusing on the former, academic medical centres, hospitals, and government agencies usually examine medical records containing information about speciality and primary care visits. ¹⁰¹ In the United States, suicide prediction is governed by the Health Insurance Portability and Accountability Act (HIPAA) which protects privacy and imposes penalties on entities when patient data is breached. ¹⁰² Research that uses patient data for suicide prediction is also subject to the Federal Common Rule, medical ethics rules and IRBs, all designed to protect human research subjects. ¹⁰³ These avenues have their own challenges. AI is autonomous, so dynamic consent may be needed from patients to use their data. ¹⁰⁴ Consent is also imperative where public data is integrated into predictive models. ¹⁰⁵

Additional considerations will arise where AI is approved for use for predicting suicide on a mental health ward in the future. Marks highlights specific risks that arise with such technology and groups those risks into concerns about safety, privacy, and autonomy. ¹⁰⁶ That framework is helpful in the inpatient context. Safety risks include false negatives and false positives. ¹⁰⁷ Those misclassifications could identify someone as being at risk of suicide (when they are not), or not at risk of suicide (when they are). ¹⁰⁸ The resulting action will be wrong for that patient. A patient thought to not be at risk may attempt suicide. A patient thought to be at risk may be detained and lose their freedoms when they are not at risk. ¹⁰⁹ Those patients may also risk having suicidal thoughts as part of their permanent medical record when it is not true. Moreover, involuntary admission in these circumstances could be traumatic and dehumanizing, raising the risk of suicide. ¹¹⁰ Improper detention or the denial of access to clinical services could form the basis of a legal claim against the clinician, the health care entity, or the developer. ¹¹¹

A potential factor in this paradigm is the complexity in mental health disorders, which have highly complex aetiology and symptomology that do not lend well to strict classification, thus, affecting the reliability of predictive algorithms. ¹¹² In practice, small sample sizes for psychiatric disorders are used which limits data for training and testing, leading to overfitting of the data and skewed accuracy measurements. Clinicians are forced to rely on less accurate models, adversely affecting suicide prediction.

While doctors should exercise their judgement, they may be reluctant to ignore the AI system because of concerns about medical liability. ¹¹³ It is widely held in psychiatric practice that suicide assessment tools should be used mainly to aid clinical judgment based on a clinical, interview-based risk assessment. However, the fear of legal consequences could result in clinicians relying more on AI than clinical judgement, which could threaten clinicians losing psychiatric risk assessment skills. Practice will vary in hospitals and jurisdictions, but the fear of liability may lead some doctors to take extra measures based on poorly assessed risks—measures that are costly in terms of the personnel required. An inpatient deemed high risk for suicide could be placed under constant direct observation, with frequent assessments being completed. That task may be cumbersome and unnecessary, but clinicians may nevertheless “play it safe” without worrying about resource implications. Thus, in a paradigm where AI exists in psychiatric wards, there are misclassification risks that could lead to staff taking inappropriate action for that patient. AI might improve risk prediction when trained properly but could provide poorer predictive capabilities than traditional methods owing to poor training or input data coupled with an overreliance and trust placed in those systems by doctors.

Privacy concerns cover data breaches and the transfer or sale of data to third parties, such as advertisers and data brokers, leading to affected individuals being stigmatized, exploited or discriminated against. ¹¹⁴ In the United States, HIPAA protects patient privacy by requiring that suicide-related data cannot be transmitted without being deidentified. ¹¹⁵ Health care providers are also prohibited from sharing non-anonymized health data with advertisers. ¹¹⁶ In England, strict rules on surveillance were noted covering the length of time that data can be stored, who can access and the restriction on third parties using that data. For AI in a psychiatric ward, strict data rules could significantly restrict many systems from being used or restrict how they handle data if they are used. If AI is used on wards, it is unclear how these factors will manifest on the ground. At present, it is hard to safeguard the confidentiality of data used in digital mental health tools due to apps being poorly understandable and developers having opaque data management policies. ¹¹⁷ Such a lack of clarity could not be permitted in the mental health setting.

Autonomy covers risks of censorship, unnecessary confinement or civil commitment, and criminal penalties or incarceration in countries where suicide attempts are illegal. ¹¹⁸ Warrantless searches could be conducted on individuals at risk of suicide based on the AI system’s prediction. ¹¹⁹ In a psychiatric ward, a patient could be moved into a locked ward based on a misclassification, undermining their autonomy, be traumatic and inadvertently increase the risk of harm. This implicates consent because confinement could occur without consent based on erroneous information. In reality, the patient is not at risk, and it would be an egregious breach of their rights if confined. ¹²⁰

Overall, AI may help to overcome some existing limitations of predicting suicide. However, its use is not without risk. Doctors may be reluctant to disagree with AI recommendations, but that reluctance could lead to inappropriate decisions for the patient, resulting in the loss of autonomy (where the AI system is wrong). Whether AI could be used in practice owing to privacy considerations restricting the use of patient data is unclear.

3.3 AI and Hospital Administration

In some hospitals, staff can be overburdened with administrative work, reducing the time for direct patient care. Bed and staff shortages often result in suboptimal risk assessments and early discharges, with subsequent high readmission rates. This part highlights these added challenges of inefficient bed management and mismatch of staff competencies to patient needs due to scheduling irregularities. It further explores the use of AI in ward administration and the legal risks associated with its incorporation in mental health.

While these administrative challenges may exist in other wards, one cannot ignore their impact in psychiatry where there is “severe underinvestment” in many parts of the world. ¹²¹ Bed shortages in mental health wards have increased over the years, and worsened during COVID-19, causing a supply-demand mismatch. ¹²² Inadequate bed management causes difficulties in procuring beds when needed, resulting in frequent patient transfers that disorient patients suffering from certain mental disorders, leading to stress and loss of trust in services. ¹²³ Moreover, the risk of homelessness and incarceration of patients with severe mental illnesses and increased risks of physical harm to other patients or members of the public, are all possible consequences of insufficient psychiatric bed capacity. ¹²⁴

Traditionally, hospital managers oversee capacity and demand based on estimations and historical knowledge. Currently, bed registries and bed tracking systems are used to track bed availability in psychiatric units. However, since the registries are not linked to EHRs or hospital admission/discharge data systems, bed tracking systems are not updated automatically. Instead, they rely on administrative and medical personnel to update manually. ¹²⁵ Time-consuming meetings are often held to discuss case management, provide updates on predicted discharges, and the current demand for beds. Information is not always up-to-date due to the staff’s focus on other responsibilities. The registries also struggle to match bed availability to patients with complex needs, such as those who exhibit violence or aggression, have co-occurring medical conditions or needs. ¹²⁶ This leads to suboptimal risk assessment, with the likelihood of patients being discharged before they receive proper treatment, resulting in increased readmission rates. ¹²⁷

Similarly, staff shortages and a mismatch of staff competencies to patient needs in psychiatric wards pose a significant barrier to treating mental health disorders due to improper utilization of staff expertise. Understaffing and poor staff rota design has led to unclear and overlapping job responsibilities, which adds to the inefficiencies in psychiatric units. ¹²⁸ According to the BMA, “staff working in mental health care are at ‘breaking point’ as they try to handle rising demand with a continuous staffing rota gap.” ¹²⁹ The copious administrative work allows little time for one-on-one direct patient care time. Long working days (of up to twelve hours) and inadequate staff scheduling have been identified as a significant threat to employee health and well-being, affecting their mental health. ¹³⁰ These rota gaps adversely impact staff training, morale, and the quality of care they provide. Consequently, medical personnel are often “forced to act above their competencies, putting patient safety at risk.” ¹³¹

While bed shortages and staff rota challenges exist in other wards, the problems in psychiatric wards are at “crisis” levels. ¹³² Hospitals, in general, are shifting towards using AI to assist with these administrative challenges. For bed management, AI is being deployed to predict discharges to free up bed space. A Portugal-based company developed an AI solution that predicts discharges by combining data about bed usage with clinical data to suggest actions five to seven days ahead. The AI system considers ward gender, surgery schedules, patient age and condition, type of bed needed, number of beds and nursing schedules, among other variables, increasing bed manager productivity by thirty to fifty percent. ¹³³ This reduces stress and enables staff to identify and prioritize the sickest patients, make faster decisions, and allocate hospital resources efficiently.

Similarly, AI is being used to predict practitioners’ workload in ICUs and EDs and ensure that the right skilled workers are linked with appropriate shifts. ¹³⁴ These systems extract information, match the competencies of health care workers to specific tasks, and help fill available slots, making the planning of rotas more efficient. AI systems are also designed to consider policy requirements when making recommendations, such as law-mandated working hours or the presence of senior doctors for specific shifts. ¹³⁵

These solutions can be implemented in psychiatric wards, albeit with caution. Developing a successful AI system for psychiatric ward administration requires high-quality data to train the algorithm and validate the model. ¹³⁶ Given that this sensitive medical data is a regulated asset, risks arise over data accuracy and privacy. ¹³⁷ A case study involving AI used to improve hospital bed allocation at Kettering General Hospital revealed the challenges of obtaining enough good quality data that cover the complexity of patient needs. These challenges include the need for the technology to be reconfigured regularly with new information about increased beds, changed ward layouts or flu admission peaks. ¹³⁸

In mental health care, the challenges of obtaining enough data and the use of EHRs to train resource allocation and staff scheduling algorithms are amplified and somewhat circular. AI requiring data on psychiatric inpatients must comply with data protections noted above. For example, in the United States, health care data in EHRs is considered sensitive, with mental health data being even more sensitive due to the high risk of stigmatization and discrimination. ¹³⁹ The HIPAA Privacy Rule provides special protection for mental health information, including extra protection for psychotherapy notes and personal notes contained within, and requires the patient’s written authorization for its use or disclosure. ¹⁴⁰

However, even if research is legally compliant, access to data may be difficult owing to the lack of data, the extra-sensitive nature of the information, and the restrictions on who it can be shared with. The quality of the data on the EHR may also be insufficient for adequately training the algorithm. Mental health information is “regularly missing from EHRs, documented in the wrong place, or under documented in specific contexts.” ¹⁴¹ This is likely due to the subjective nature of information, ¹⁴² greater reliance on narrative progress notes as opposed to quantitative measures, ¹⁴³ and stigma surrounding mental health conditions. ¹⁴⁴ Fragmented use of paper and electronic records also contributes to missing mental health information on EHRs. ¹⁴⁵ This ambiguous and, at times, incorrect documentation of mental health data may be insufficient to train AI systems for the administration of psychiatric wards. Thus, even if developers can overcome the legal challenge surrounding privacy, they may be limited by the data that is available.

4. The Need for Guidelines

The analysis above reveals that although AI-based solutions have shown promise for monitoring patients, predicting illness, and assisting with hospital administration, the application of the technology in psychiatric wards may be challenging. The risks associated with adopting such technology require serious scrutiny, given the capacity of AI recommendations to aggravate existing issues or to provide limited assistance. The legal and ethical implications highlighted surrounding patient privacy, informed consent, access to accurate data, and data protection and privacy might curtail the adoption of AI systems in the psychiatric setting. This part further elaborates on these considerations and calls for guidelines to be developed that account for those risks should AI systems eventually be used in psychiatric wards. ¹⁴⁶ Some AI systems may never gain approval for use, but the purpose here is to provide pre-emptive baseline considerations for stakeholders instead of reactionary measures post-implementation.

First, a major concern is the loss of privacy for patients. For example, delusions of persecution, in the form of camera monitoring, are among the most common delusions in mental health services. Many patients perceive being subjected to constant monitoring as dehumanizing, and continuous surveillance increases stress and anxiety levels in patients and decreases trust in those around them, ¹⁴⁷ which may worsen their existing mental illness and make them more resistant towards adopting surveillance technology for their care. ¹⁴⁸ Guidelines for AI video surveillance should be adopted to protect privacy by mandating that patient data should be used for a specified purpose and shared with only a narrow range of people, such as their doctor on the ward or the developers, to ensure the AI system operates properly. Smart cameras can be configured to employ privacy mask applications to hide or blur out faces in videos or anything else that might reveal a person’s health information, such as computer screens displaying patient details. ¹⁴⁹ This would allow enhanced monitoring of psychiatric patients in their rooms while lowering the risk of violating their privacy and ensuring that their sensitive health information remains secure. From a research perspective, any research must comply with heightened norms and protections surrounding mental health data. However, data is crucial to training effective algorithms. One potential solution could be to develop algorithms that can be trained on anonymous pixels instead of patient images. ¹⁵⁰

Second, underlying the privacy concerns is the need for consent for using AI systems in psychiatric wards. While patients’ consent to being monitored by AI in a public ward is already a contested issue, it is further complicated in a psychiatric ward where patients are more likely to be admitted involuntarily. ¹⁵¹ The stigma associated with mental illnesses and its potential effect on a patient’s professional and personal life – work opportunities or marriage prospects, for instance – marks a significant challenge for obtaining consent for using AI systems in that context. Furthermore, even if consent were obtained for AI, that consent might be rendered void where an AI system evolves by making recommendations beyond its initial intended purpose that the patient agreed to.

Adequately informing the patients and hospital staff when using an AI system and the type of data these systems have access to is not enough. Approval may be required from the courts for using or affirming the use of AI systems on the ward and be subject to ongoing review by a judge. A corollary challenge is explaining the reasons for AI recommendations (explainability and the black-box problem) to doctors and patients. ¹⁵² As such, future guidelines should require that the relevant legal approvals be obtained where necessary for the use of an AI system and that those systems should be explainable so that stakeholders can understand the capabilities, scope of use, and limitations of these systems. ¹⁵³

Further, should consent challenges be overcome, it may be necessary to implement a flexible approach for obtaining consent in practice, owing to the ever-evolving nature of AI systems. One option could be electronic informed consent to improve individual control and choice. ¹⁵⁴ It would allow users to understand the consequences of consent by determining which data will be used and how. Given the frequent updates of AI software, this approach can be enhanced by incorporating “dynamic consent,” allowing patients (with capacity) to modify their consent periodically depending on the use of the information they wish to permit. ¹⁵⁵ A framework that allows patients to opt out of constant surveillance could also be considered. Taking Eindhoven-based Catharine Hospital, for instance, the hospital and researchers obtained explicit consent and designed the smart camera with a switch to turn off the device if a patient did not wish to be monitored.

Third, considerations around data security, privacy and algorithmic accuracy arise. For example, health data collected by AI may exceed what is required and may later be repurposed for uses that have serious ethical, legal, and human rights implications. ¹⁵⁶ Massive amounts of mental health data are required to train AI algorithms for suicide prediction and for the effective administration of psychiatric wards. Policies regulating such data collection should address mechanisms aimed at preventing the “collection of illicit or unverified sensitive data.” ¹⁵⁷ Given this sensitivity of data, clear guidelines are needed highlighting the laws that must be followed when handling such data in these and other scenarios. ¹⁵⁸ Clear frameworks for how medical practitioners and researchers use data while safeguarding patient confidentiality will promote public trust, advancing the use of AI in the health care setting.

5. Conclusion

Overall, the potential use of AI in psychiatric wards raises challenges that are distinct enough within the mental health domain to warrant closer consideration. The examples of AI integration into surveillance systems, suicide prediction, and hospital administration highlight those specificities concerning patient privacy, informed consent, and data protection, among others. While those concerns may also arise in other hospital wards, additional factors apply in the mental health care arena owing to the greater sensitivity of the information there. For that reason, laws, norms, practices, and protections have developed to protect psychiatric patients – factors that ultimately trickle down into considerations about using AI in psychiatric wards.

The examination in this Article yields two pertinent insights. First, should AI be approved for use in systems on psychiatric wards, any such approval would likely be contingent upon significant restrictions regarding how those systems will be used and monitored. Further, if those systems are used, they will likely be on an individual patient basis and be subject to frequent review where a legal question arises. Second, specific guidance on the legal considerations that arise would be beneficial for hospitals, patients, and the courts to establish best practices for using and implementing AI on mental health wards. Without such guidance, risks may arise of different hospitals using and implementing such systems incongruently, resulting in diverging levels of protection for patients. That ought to be the next contemplation for research in this area, supported by interdisciplinary efforts between hospitals, developers, lawyers, and bioethicists.