The Commission for Health Improvement's challenge

The Commission for Health Improvement (CHI) has been assigned the formidable challenge of auditing the implementation of National Service Frameworks (NSFs; Department of Health, 1997). The coronary heart disease NSF will be reviewed first, and this is likely to be followed by the mental health NSF. If the CHI is to accomplish its task, it must successfully negotiate two hurdles: the existing arrangements for reviewing NHS services and the limitations of available data. The enormity of this task cannot be overestimated and is particularly well illustrated by considering the mental health NSF.

Entering a crowded field of mental health service regulators

Mental health services are currently scrutinised by three organisations (Department of Health, 1999). First, the Audit Commission has been running a Value for Money Audit, following publication of its national report on mental health services for older people (Audit Commission, 2000). It plans to continue its interest in NSF-related areas. Second, the Social Services Inspectorate has three tranches of mental health service inspections planned between now and 2004. Third, NHS regional offices receive a steady stream of data from their health authorities concerning the progress made in implementing the NSF. All these organisations have extensive and different experiences in both the development and evaluation of inspection methods and in the recruitment and training of inspection teams. They have developed relationships with health service providers, who know what is expected. In such a crowded field, it is not unreasonable for the three established regulators to expect to collaborate with the CHI to ensure that the development and implementation of the NSF review is informed by their experiences.

Limitations of the data

The second hurdle to be overcome is the development of an inspection package that will accurately assess the implementation of the Seven Standards set out in the mental health NSF, and that will provide a comprehensive reflection of the extensive activity that has taken place around the country in response to the NSF. This is an enormous undertaking for two reasons. First, in some cases the indicators or milestones suggested won't actually measure implementation of the Standard. For example, Standard Two states that anyone who contacts primary care with a common mental health problem should have his or her needs identified and should be offered any necessary treatments. However, suggested indicators, such as improvements in the psychological health of the population as measured by the National Psychiatric Morbidity Survey, are too insensitive to reflect implementation of the NSF (Jenkins & Meltzer, 1995). The second problem arises where the indicators or the data sources should, in theory, be able to demonstrate implementation of the standards, but, in reality, currently available data are unlikely to be detailed enough, complete or accurate. For example, Care Programme Approach (CPA) returns are often cited as a data source, but their ability to act as such depends on their content and accuracy, qualities that vary across trusts. Even when integration of care management and the CPA are fully established, the routine data they generate will not be sufficient to demonstrate successful implementation. This is because the data only provide information that a care plan exists, rather than evidence as to actual ability of users to contact such services.

The way forward

The challenge for the CHI review team is to negotiate these hurdles in a sensitive but thorough manner. To achieve this, the CHI must work jointly with the other regulators: the Audit Commission, the Social Services Inspectorate and NHS regional offices. The CHI, in collaboration with these organisations, must critically review the strengths and weaknesses of: current inspection methods and the processes used to review the reliability and validity of their findings; the routine data currently requested; their methods for involving users and carers; and recruitment and training strategies. These issues need to be sorted out now so that they can inform the rolling programmes currently underway and so avoid duplication of effort by the providers of care.

This joint team must then analyse in detail each standard, its indicators and currently available data as a first step towards developing an inspection package. Such a package should be developed in collaboration with mental health professionals, users and carers. The joint team's objective should be to examine the extent to which each of the standards are, where possible, being implemented so that users and carers are involved in the choice and delivery of effective, acceptable, appropriate, accessible and safe care. Both quantitative and qualitative data sources will be needed, including local strategic documents; audit and financial data; questionnaires and semi-structured interviews; and original research. For example, assessment of the implementation of Standard Two ideally requires a population-based survey of the level of clinical need compared with the level of detection and the type of intervention. This research is beyond the remit of the CHI, but could be carried out by research institutions. Current research suggests that some groups, such as ethnic minorities, do not have their mental health needs met (Raine et al, 2000; Commander et al, 1997). Repeating these studies in the next few years would provide one indication of the impact of the NSF, and any such studies should therefore be identified by the joint team as part of the review. In addition to the data suggested in the NSF, the joint team should seek out local primary care audit data on the management of patients with mental health needs (or plans to undertake audit). This will provide information about patients with mental problems who have not been referred to specialist services, and about the range of specialist services used. Finally, service providers should be asked to provide information about intended and actual implementation of schemes designed specifically to address this standard.

The adoption of such an approach will ensure the production of a methodologically robust audit. It is therefore reassuring to know that joint working between the CHI and the Audit Commission, Social Services Inspectorate and the NHS Executive has been agreed, and that a detailed analysis of data needs is planned. This preparatory work could take well over a year to complete, but it should mean that the review of the implementation of the NSF will be accurate, fair, and comprehensive.