From CHI to CHAI: what a difference an ‘A’ makes

When the National Health Service (NHS) was born in 1948, there was no specific agenda for clinical quality, which was assumed to be inherent in the system. The received wisdom was that good professional training with self-regulation and appropriate opportunities for ongoing professional development all inevitably added up to high-quality health care. Unfortunately, as we now know, this was not and is not true. What quality initiatives there were tended to be disconnected, with duplication, inefficiency and complicated processes all too often the result. Understanding of the relationships between structures, processes and outcomes improved in the 1970s, but in the 1980s, effort was arguably constrained by a prominent emphasis on organisational performance and cost containment.

Not surprisingly, staff delivering treatment and care in the NHS were often painfully aware of the failings and frustrations of the service, but it took a number of high-profile scandals in the 1990s (such as children's heart surgery in Bristol) for clinical quality to receive the attention it needed. It became ‘a prevailing purpose, no longer a desirable accessory’ (Department of Health, 1997) and was enshrined as a statutory duty for trusts in the form of clinical governance.

As clinicians, this is what we had been calling for. We had long been arguing that what happened to patients was more important than what happened to finances, but opinions were divided. Details of how it would all work emerged the following year (Department of Health, 1998) and inevitably an inspection system was to be a key component. The notion of more regulation and inspection was treated with caution and suspicion in some quarters and in mental health services questions were raised about the number of inspections, and indeed the balance between benefits and costs (Goldberg, 2002). But the Commission for Health Improvement (CHI) came to pass and soon it will pass away, to be replaced by the Commission for Healthcare Audit and Inspection (CHAI). What, therefore, have we learnt from CHI about the quality of treatment and care in mental health services, and what will the future look like?

The Commission for Health Improvement

CHI was described as the ‘independent authoritative voice on the state of the NHS in England and Wales’ (www.chi.nhs.uk). One of its key functions has been to undertake clinical governance reviews of NHS acute and specialist trusts, mental health trusts, NHS Direct sites, ambulance trusts, primary care trusts in England and local health boards in Wales. Review teams have been recruited through national advertising and each has included at least a doctor, a nurse, an NHS manager, a lay member and an associated health professional. The intention has been to review the effectiveness of each organisation's clinical governance arrangements, and identify and share examples of best practice together with areas for improvement.

During its 4 years of operation, CHI's methodology and construction of its teams have been the subject of debate and some criticism, largely centred on the relative lack of review experience and questions around the conclusions drawn from the sampled data (Burns, 2002). However, CHI's response has been to modify its methodology over time and assert that its assessment frameworks enable reliable and consistent assessments (Patterson & Cornwall, 2002). In its 4 years, CHI has reviewed over 300 NHS organisations and one of its parting shots has been to produce a report on its findings in mental health trusts (Commission for Health Improvement, 2003).

CHI Sector Report - Mental Health Trusts

This report makes interesting reading for a number of reasons. It says a lot of things that have apparent face validity (which we think we know to be probably true), but there is a disappointing lack of data and firm evidence. To begin with, it states it states that while the direction of travel in mental health services appears to be right, future quality improvement is likely to be constrained by ‘serious’ lack of capacity. In addition, CHI has found ‘considerable dissatisfaction and frustration’ among mental health services that they are not treated as a real priority in practice. CHI claims to have identified common factors in trusts that do well in their reviews and those that do not. These are summarised in Table 1.

Table 1. Shared characteristics of high-performing and low-performing trusts in clinical governance reviews

Characteristics shared by trusts performing well in clinical governance reviews	Characteristics shared by trusts performing poorly in clinical governance reviews
Lower vacancy rates, particularly in psychiatry, or active attempts to resolve vacancy problems; high staff morale; good progress with Improving Working Lives	Serious problems with recruitment generally in psychiatry and in-patient nursing; low morale; cultural and operational divide with social care staff
Good progress with developing national service frameworks/NHS Plan services and the care programme approach	Limited or partial developments of new services and limited implementation of the care programme approach
Leadership is cohesive, visible and well-regarded by staff and partners	Staff perceive leadership as remote; weaknesses in executive or non-executive leadership
Strong relationships between clinicians and managers	Lack of engagement of clinicians in management
Cohesive structures between different parts of the trust	Disconnection between different parts of the trust
Strong structures to support clinical governance in directorates and sectors/localities; understanding of relationships between the board and directorates, sectors and services	Limited structures below corporate level to support implementation and performance management of clinical governance, or structures to support clinical governance components
Well-developed clinical information systems and progress with performance management	Fragmented information systems and little development in performance management
Good progress on organisational and operational integration with social care	Limited progress with organisational and operational integration with social care
Effective communication systems in place	Poor communication systems

It is difficult to know what to make of this. Although none of it is surprising and it makes intuitive sense, without a clear idea of how these dimensions are benchmarked, it is arguably impossible to know where any individual organisation is on each given continuum.

The report tells us that there are ‘serious problems’ with recruitment and retention in mental health, which we have known for some time, but it does make the point that an organisation struggling to maintain a service is unlikely to prioritise the necessary systems for improving the quality of care. Again, this is not surprising but it does highlight the fact that recruitment, retention, training and new professional roles must be among the highest priorities to consider in mental health policy.

Investment in information systems has undoubtedly lagged behind in mental health services and the report describes capacity as being ‘severely under resourced’. Certainly, effective clinical information systems that tell us what we want to know about our patients are essential if we are to be able to answer the questions we need for effective clinical governance and quality improvement. In addition, access to integrated records on multiple sites would significantly improve our ability to deliver the right treatment and care and reduce risk. The NHS programme for IT is therefore to be welcomed as long as it delivers this, and soon.

CHI is ‘disappointed’ with risk management in mental health. It says that, despite good systems for incident reporting and review, they found little evidence of feedback to staff and inconsistent approaches to violence and aggression ‘in a number of trusts’. Concern is expressed about drug administration and ‘inconsistent’ processes for risk assessment. Nevertheless, it is not clear why (or indeed if) the safety concerns are significantly different from the rest of the NHS, especially as risk assessment and management have been such prominent concerns in mental health services for so many years.

The sector report contains a large section on the user perspectives highlighting access, in-patient environments and activities, seclusion and a lack of priority given to services for older people. It also raises the disadvantage experienced by black and ethnic minority service users and makes the point that good practice in working with specific communities in different ways is not effectively shared across services.

Overall this is, in my view, a disappointing document in that although it undoubtedly highlights areas in which mental health services are deficient (or indeed doing well), the analysis, such as it is, is insufficiently rigorous to provide us with powerful arguments for sustained additional investment. So as the curtain closes on CHI, what might we expect from its successor?

The Commission for Healthcare Audit and Inspection

CHAI is a brand new organisation and it replaces CHI, the National Care Standards Commission (the part of it which inspects private and voluntary healthcare organisations) and also takes over the studies of efficiency, effectiveness and economy of healthcare previously done by the Audit Commission. In due course, it is expected to incorporate the Mental Health Act Commission and be involved as an independent stage of the NHS complaints procedure.

Quite what the style of this organisation will be is unclear at the time of writing, but according to the vision document circulated for consultation it will ‘provide patients and users of services with clear assessments of the safety, quality, efficiency and effectiveness of the services that they receive’ and will ‘develop, with others, clear criteria and measures in the light of which standards of healthcare will be assessed’ (Commission for Healthcare Audit and Inspection, 2003). It also says that it will ‘treat all individuals and organisations fearlessly and even-handedly on the basis of robust evidence which all those affected by it have had an opportunity to see and respond to’.

CHAI also says that it will seek to reduce the burden of regulation on providers of health and social services, and that it will seek to consolidate the collection of data. It intends to work with the Department of Health, the Royal Colleges, the NHS Information Authority, the NHS Litigation Authority and a wide range of others to agree a template of common data required and the form in which it should be collected. This would certainly be a significant step forward.

CHAI's vision and priorities as currently stated incorporate an emphasis on improvement and ‘information based assessments’. The four steps articulated to create a framework for performance assessment for the NHS are:

1. • The development and publication of performance standards. These are for the government to determine (and will include existing national standards enshrined in national service frameworks), although it is clear these will be developed and refined over time.

2. • The development of criteria for assessment against these standards. These are for CHAI to determine and wide consultation is to be anticipated.

3. • Devising appropriate questions and indicators to assess whether criteria have been satisfied. Again, it is to be expected that these will be subject to wide consultation.

4. • The identification of appropriate data to make up ‘intelligent information’ in order to make appropriate judgements. Data currently collected may go some way towards this, although it is likely that new information and new ways of collecting it will need to be devised over time. It is unclear whether any existing data collection will be dropped.

CHAI also describes the development of a ‘local presence’, with targeted and proportional visits, in an attempt to minimise the burden of regulation. Quite what this will look like is unclear, but there is an expressed intention to develop partnerships with other regulators, which would surely be welcome.

And so?

And so CHI passes into history and we move into an uncertain future for regulation and inspection. The debate still goes on about the best way of improving standards and driving up the quality of our services, but we must work with what we have if we are to get the best we can for our patients.

Despite some of the criticisms of CHI's sector report on mental health, there do seem to be some useful lessons to be extracted, and it certainly highlights the relative under-investment in mental health services and raises questions as to where the promised extra money has gone. This is especially important now, given the risks posed by a potentially unstable financial environment created by the devolution of commissioning to primary care trusts.

If we are to secure adequate investment in the future, however, we need good analysis relating resources (existing or additional) to outcomes, which requires us to ensure that we measure the right things. Crucially, this needs clinicians to be actively involved in developing measures of clinical quality and to participate constructively in the debate about their application. If we do not do this, we run the risk of jeopardising future investment in mental health services.

We cannot assume that passionate commitment to patient care and a keen sense of fairness will win the day. Without the necessary resources we will continue to struggle to provide the quality of care and treatment we would all like to see. Or to quote Dr James Reinertsen, ex Chief Executive of the Beth Israel Deaconess Medical Centre and a close friend of Don Berwick, ‘If you are fighting for air and water, it's kind of hard to be doing opera’. Quite.