The aim of this editorial is to explore what we can learn from international approaches to the measurement of quality of mental health care. The salient and timely measurement of high-quality mental health care has the potential to play a critical role in improving mental health systems, yet it is not without challenges. Lessons from international collaboration – of which there are numerous examples – can inform this work. Of utmost importance is that efforts to measure quality of mental health care result in quality improvements. This in turn can provide an impetus to improving and expanding efforts to measure and monitor the quality of mental health care. This paper outlines: why it is important to measure the quality of mental health care; challenges in measuring quality of mental health care; the importance of developing a measurement infrastructure; and the factors that should be considered when developing mental health quality indicators. It then summarizes international endeavours in this area; some potential uses of mental health quality indicators; and considers whether measurement of quality of care can translate into improvements in quality. The paper concludes with a brief discussion.
Importance of measuring the quality of mental health care
Overwhelming international evidence suggests health care is often not delivered in accordance with evidence-based and commonly-agreed professional standards, resulting in poor quality and unsafe care that harms tens of thousands of people every year, and the squandering of scarce health care resources (OECD, 2010a). Unfortunately mental health care is not exempted from this problem, with far-reaching and considerable consequences for individuals, families and society. Neuropsychiatric disorders are a leading cause of disability accounting for 13% of the global burden of disease worldwide (World Health Organization, 2008) translating into a high economic cost to society. A conservative estimate from the International Labour Organization estimates that the costs of mental ill-health account for 3–4% of gross domestic product in the European Union (EU). Moreover, the majority of these costs fall outside the health sector. Mental illness is responsible for a very significant loss of potential labour supply, high rates of unemployment and a high incidence of sickness absence and reduced productivity at work. The early onset of mental disorders can lead to decreased academic achievement for children, teenagers and young adults (OECD, 2012). Mental disorders also lead to increased demands on other sectors such as social welfare and criminal justice. Thus, it is essential that all people with mental disorders can access and use high quality and effective care as poor care can hinder improvement and recovery (Institute of Medicine, 2006). Deficiencies in the quality of (mental) health care can result at all levels of care and cross-sectors. There may be problems in the organization of care, access, capacity, poor co-ordination and poor decision support for clinicians (e.g., due to a lack of clinical guidelines). Poor care may also be due to factors outside the immediate care delivery environment, such as policy, payment and regulation. The grave consequences of poor quality and ineffective care create an impetus to improve measurement and ultimately performance (Institute of Medicine, 2006).
Challenges in measuring the quality of mental health care
Defining and agreeing what is high quality of care and determining which salient measures can capture this concept is not a simple task. Mental health problems can be complex and multifactorial in nature and thus can require multifaceted interventions and approaches involving different agencies and sectors. Critically defining high-quality care involves having an agreed conceptual framework with users and carers, as well as professionals. Co-morbidity between mental disorders and substance abuse and physical ill-health are common. The frequently found separation between physical and mental health care systems can lead to discontinuities and a lack of co-ordination of care. It can also make defining and measuring quality more difficult. Mental health service users may find it more difficult to participate in shared decision making as a result of the problems they are experiencing and importantly as a result of staff attitudes and behaviours. This may lead to under reporting of adverse effects. Involuntary treatment not only has implications for patient decision-making and control over care, but also makes measuring performance and quality of care vital and more complex (Pincus et al. Reference Pincus, Spaeth-Rublee and Watkins2011). Delivering high quality cross-sectoral care is potentially dependent on several agencies and professionals working together. For example, health services may work with other sectors, such as social welfare services, employment, education and housing. Developing and measuring salient quality outcomes within this complexity are difficult. There are inter-dependencies between these different sectors in terms of their impact on mental health outcomes and quality of care, which may make it more difficult to establish accountability when short-comings arise.
Quality measurement infrastructure needs to be further developed and utilized
Mental health care suffers from a less developed quality measurement infrastructure than general health care (Institute of Medicine, 2006). Information technology tends to be under-developed and less widely adopted and used for clinical care support. There remains a need to build information systems into psychiatric practice (Harding et al. Reference Harding, Rush, Arbuckle, Trivedi and Pincus2011) and address weaknesses in current data systems that hinder the reporting and monitoring of quality of care measures (Herbstman & Pincus, Reference Herbstman and Pincus2009).
In 2008, the Organization for Economic Co-operation and Development (OECD) assessed the availability of information to measure and compare quality of mental health care across OECD countries (Garcia-Armesto et al. Reference Garcia-Armesto, Medeiros and Wei2008). The study found that the data sources most widely available across countries at the time were hospital administrative databases, national surveys and national registries. The absence of a unique patient identifier (UPI) in many countries posed problems for constructing certain quality of care indicators, in particular those assessing continuity of care and quality of prescription or treatment in primary and community care. These are the settings where an increasing proportion of mental health care is provided in many OECD countries. Moreover, the study revealed the low level of integration of information systems across different levels of care provision, which again has implications for assessing continuity of care.
More recent OECD work in this area reviewed the extent to which 20 countries participating in the OECD Health Care Quality Indicators (HCQI) project have national mental health in-patient data and the infrastructure in place to support data linkages (Oderkirk, Reference Oderkirk2012). Sixteen countries reported mental hospital inpatient data are available at a national level, whereas fourteen of these countries reported using mental hospital in-patient data to regularly report on health care quality. Eleven countries reported that national mental hospital in-patient data contains a UPI number that could be used for record linkage. Five countries reported that national record linkage projects were used for regular mental health care quality monitoring. These results illustrate that there is still some progress to be made in terms of having a comprehensive and well-designed infrastructure in place to measure the quality of mental health care across OECD countries. It is also worth emphasizing that mental hospital in-patient data are generally better developed than data for community and other mental health care.
Considerations for constructing quality of care indicators
In order to construct indicators to measure quality of mental health care it is useful to take into account seven fundamental elements outlined by the Institute of Medicine (2006).
Conceptualizing the aspects of care to be measured: Pincus & Spaeth-Rublee (Reference Pincus and Spaeth-Rublee2012) and Kilbourne et al. (Reference Kilbourne, Keyser and Pincus2010) posit that quality measures must meet several criteria in terms of selection and evaluation, namely clinical importance, validity and feasibility. Quality of care measures must be clinically important so that providers can improve performance on the measure and such improvements will make a difference to overall quality of care. Measures must also be important from the perspective of service users and carers. Measures must be valid or scientifically sound. If the measure may be affected by the severity or complexity of the population under treatment then there should be methods to risk-adjust the measure. Finally measures must be feasible to collect and not impose a high burden on the collecting entity in terms of cost and time. A further and fundamental challenge faced at this initial stage is that of achieving consensus across all stakeholders on the measures chosen.
Translating quality-of-care measurement concepts into performance measure specifications: this requires the specification of well-defined numerators and denominators along with data sources; inclusion/exclusion criteria; and time-frames for data capture.
Pilot-testing the performance measure specifications: in order to determine validity, reliability, feasibility and cost of collection.
Ensuring calculation of performance measures and their submission to a performance measurement repository: it has been shown that successful quality initiatives in general health care have taken place when there is (1) a critical mass of influential supporters or individuals committed to either requiring or carrying out the calculation and submission of measures; or (2) ongoing commitment of sufficient resources to enable the analysis of quality measures, thus making such measures useful so that those calculating and submitting them do so voluntarily.
Auditing to ensure that performance measures have been calculated accurately and in accordance with specifications
Analyzing and displaying performance measures in formats that are understandable by multiple intended audience(s) in order to ensure their usefulness for multiple stakeholders. The particular avenue chosen to analyze and display measures will also depend on the ultimate use or aim of the measures e.g. public reporting, benchmarking or as part of a provider payment scheme.
Maintaining effectiveness of performance measures and policies over time: the specifications of measures may change over time, perhaps due to changes in coding practices or health systems or because of unanticipated issues in the measures themselves.
International initiatives in measuring the quality of mental health care
It has been argued that quality improvement is and should be an international endeavour to facilitate learning and mental health care should not be an exception (Mainz et al. Reference Mainz, Bartels, Rutberg and Kelley2009). Frameworks, terminologies and definitions, as well as philosophies and methods, designs and principles should be shared internationally. There have been several cross-country initiatives focusing on collecting information pertinent to the quality of mental health care, which are briefly outlined below.
Organization for Economic Co-Operation and Development
HCQI
The OECD HCQI project was launched in 2003 and has since developed and tested a range of internationally comparable health care quality indicators covering various health care domains including mental health. An international expert panel recommended a set of 12 indicators covering 4 domains of continuity of care, coordination of care, treatment and patient outcomes. Subsequent work assessed the availability of information systems to collect these indicators. They found that hospital re-admissions for psychiatric patients, mortality for persons with severe psychiatric disorders and length of treatment for substance-related disorders were the most viable indicators immediately available to start the data collection. Currently, only 2 of the original 12 indicators are collected: 17 OECD countries collect data on re-admission rates for schizophrenia disorder and 18 OECD countries collect comparable data for bipolar disorder.
“Mental Health Systems in OECD countries” indicator benchmarking club
The “Mental Health Systems in OECD Countries” was launched in 2011 to analyze several aspects of OECD mental health systems including quality of mental health care. The benchmarking club comprises a number of OECD countries who have indicated an interest in working together to try and establish whether they could compare performance across a small number of mental health quality indicators. This activity has several aims including:
• the sharing of good and innovative practice,
• learning from successes and failures within each country, and
• agreeing a small set of salient mental health quality indicators for meaningful comparison across the participating countries. Potential measures include mortality from suicide in people in contact with mental health services, involuntary admissions, and use of seclusion and restraint.
International Initiative for Mental Health Leadership (IIMHL) Clinical Leads project
In 2008, clinical experts from 12 countries, meeting as part of the Clinical Leads group of the IIMHL, initiated a project in order to develop a consensus framework for reporting on mental health care quality. The ultimate aim of this project is to develop and implement a balanced, inclusive and common framework of measures that will allow for international comparisons and benchmarking of system performance, with a long-term goal of informing initiatives that will improve mental health services in these countries.
The work for the IIMHL Clinical Leaders Group project has consisted of a systematic review of peer-reviewed journal articles, government reports, white papers and other grey literature on population-based performance measurement in mental health in the 12 countries and the compilation of indicators collected through this literature review. In addition, a survey instrument has been developed to identify core set of indicators collectable across countries.
Nordic Indicator Project
The Nordic Council of Ministers initiated the Nordic Project on Measuring the Quality of Health Services in 2007. The project group was asked to prepare proposals for indicators that could comprise the basis for registering and monitoring the quality of mental health services in the Nordic countries (NORDEN, 2011). The project group decided to include all indicators of quality that are currently used to indicate the quality of mental health services in the Nordic countries in relation to hospital (inpatient and outpatient) treatment, as well as potential indicators that are expected to be implemented in the Nordic countries in the near future. The Nordic countries have unique opportunities to measure the quality of mental health services because of well-established health-related registries and because data can be collected that are linked to individual patients. Thus, the Nordic countries can contribute substantially to inspiring and collaborating on the international measurement of the quality of mental health services based on quality indicators.
REFINEMENT
The REFINEMENT (REsearch on FINancing systems' effect on the quality of MENTal health care) project consists of collaboration between research institutions in nine EU countries. The activities of the work package on quality of care and met/unmet needs include a literature review on indicators of quality of care, an assessment of the availability of data on performance and outcome indicators of quality of mental health care and the collection and analysis of data on accessibility, policy for equity, assessment of needs, mortality, quality-of-life and satisfaction.
However, the undertaking of such projects is not without the challenges inherent in international benchmarking as outlined by Mainz et al. (Reference Mainz, Bartels, Rutberg and Kelley2009). These challenges include: the difficulty of collecting data for even relatively simple indicators; reported indicator data may be related to different years for different countries; the differential ability of countries to track patients after hospital admissions, which is related to the presence or absence of UPIs; the lack of risk adjustment; deficiencies in relation to validity testing and exhaustive and exclusive data specifications; and the inability to avoid choosing indicators for quality benchmarking that reflect the data source that is available rather than optimal measures of quality of care.
Use of quality indicators for mental health care
There are many potential uses of quality indicators including: benchmarking, performance management and quality improvement; consumer information; and provider payment. Quality indicators can be used for benchmarking purposes at both a national and an international level. Public reporting of quality metrics engages providers and organizations to improve performance and enhance their reputation. It is also crucial for holding health care organizations accountable for improving care (Kilbourne et al. Reference Kilbourne, Keyser and Pincus2010). New methods of paying providers to improve the quality of health care, often known as pay for performance (P4P) are becoming increasingly more common in OECD countries. Traditional methods for paying physicians such as salary, fee-for-service or capitation pay for quantity not quality. As these schemes are new and many have limited formal evaluation, the evidence at the moment is still insufficient to draw definitive conclusions. However, the experience to date suggests that it is possible to improve quality and efficiency by paying for it. One of the critical issues that have afflicted P4P programmes is monitoring the quality indicators as in many instances information on quality is not routinely collected by physician practices (OECD, 2010b). At present, indicators pertinent to mental health are not pervasive in P4P programmes (Harding et al. Reference Harding, Rush, Arbuckle, Trivedi and Pincus2011).
Does measurement of quality of care translate into improvements in quality of care delivered?
To paraphrase Lord Kevin ‘You can't improve what you can't measure’. At the clinical level, introducing and applying standardized longitudinal measurement-based care for clinical evaluation and treatment is a key strategy to improve the quality of mental health care (Pincus & Spaeth-Rublee, Reference Pincus and Spaeth-Rublee2012). However, it is also necessary to link quality measurement with activities at the centre of care and the day-to-day operations of providers in order to fully effect change. Communication, training, stakeholder involvement and use of mechanisms for measurement, feedback and redesign are all essential to successful implementation of new and changing work practices to improve quality of care (Institute of Medicine, 2006). Clinicians at the front line need to be actively engaged in the process of improving quality at multiple levels (e.g., payment, licensure and organizational change), and payment systems should reward high quality of care. The adoption and use of measurement-based care practices should also be an integral part of clinicians training and education (Harding et al. Reference Harding, Rush, Arbuckle, Trivedi and Pincus2011). Standardized measures that allow results to be monitored and tracked uniformly over time are the foundation of performance improvement. Finally, measures should be used by multiple stakeholders at multiple levels to actually improve care (Kilbourne et al. Reference Kilbourne, Keyser and Pincus2010).
Concluding remarks
The importance of measuring the quality of mental health services is increasingly recognized. There are still significant hurdles to be surmounted, perhaps the most vital of which is the development and improvement of the infrastructure necessary for measurement and reporting quality. It can be expected that the impetus to overcome these challenges will gain momentum as it becomes more and more evident that quality measures have many valuable purposes and outcomes, not least of which is better mental health care.
Declaration of interest
None
