Progress on NICE guideline implementation in mental health trusts: meta-analyses

Abstract

Aims and Method

To investigate implementation of National Institute for Health and Clinical Excellence (NICE) guidelines in mental health, focusing on the schizophrenia guideline. Data analyses centred on implementation of the guideline, as well as looking at a set of markers mapped to the NICE principles of implementation and other identified clinical prerequisites. A self-report questionnaire tool was sent to senior executives at mental health trusts containing questions linked to the markers of implementation and clinical prerequisites; responses were analysed with data from the Healthcare Commission audit of implementation of the guideline to show key relationships.

Results

Information from both data-sets (senior executive data collection and the audit) showed that implementation is patchy, with pockets of good implementation. Findings indicate that higher levels of implementation are linked to corporate commitment and leadership, as well as support from commissioners.

Clinical Implications

Implementation might be improved by corporate commitment and leadership and better support from commissioners.

To date there is little research about how successfully trusts are implementing guidance from the National Institute for Health and Clinical Excellence (NICE), but what there is suggests that implementation is inconsistent (Sheldon, 2004). A large amount of information on evidence-based implementation methods is published by NICE to support trusts with this task (National Institute for Health and Clinical Excellence, 2006). In order to determine the degree of progress and the key implementation processes used, the National Collaborating Centre for Mental Health (NCCMH) carried out a survey of mental health trusts, analysing data alongside findings from the Healthcare Commission audit of the implementation of the schizophrenia guideline.

Method

We used data from two sources to unpack progress with implementation: data from a bespoke questionnaire tool, and a meta-analysis using this alongside data from the Healthcare Commission audit of the implementation of the NICE schizophrenia guideline. The questionnaire items were derived from two sources, to answer two key questions: first, how far have trusts followed NICE's principles of implementation; and second, what evidence is there that trusts are implementing guidelines? Question one is answered by assessing the extent to which trusts gave positive responses to specific questionnaire items derived from the NICE principles. To answer the second question, expert advisors to the project have identified key implementation markers, which show that a trust is fully engaged with implementation. Questionnaire items were mapped to the principles summarised in Box 1. The forms were sent to all chief executives and medical, clinical and clinical governance directors at all National Health Service (NHS) mental health trusts (including primary care trusts providing mental health services) in England as well as combined trusts in Wales (a total of 488 questionnaires sent to 79 trusts).

Data from the recent audit of implementation of the schizophrenia guideline (Healthcare Commission, 2006), carried out as part of the Healthcare Commission joint review of community mental health services, were re-analysed alongside data from the present study. The audit identified a number of key implementation markers, and gathered data to assess adherence with these: availability and use of family interventions and cognitive–behavioural therapy (CBT), provision of information and occupational needs, physical health checks, prescribing (atypical antipsychotics, depot formulations, treatment-resistant schizophrenia), monitoring, medication and advance directives. A proxy measure of guideline implementation was created within the data-set by summing the trust-level scores for each of the indicators in the audit.

Data were analysed using SPSS version 14 for Windows. Correlations and one-way analyses of variance (ANOVAs) were used to identify significant univariate relationships; linear regression was used for multivariate effects. In order to examine fully the effect of corporate commitment and leadership, a new variable was constructed comprising identified executive lead, identified clinical director lead, clinical champion, team to support the clinical champion, and reports to trust board. The new variable showed an acceptable Cronbach's μ value of 0.68.

Results

A total of 209 usable forms were returned, an adjusted response rate of 47% (209/(488–44)). Data are available for the various professional groups sampled: 59% of forms were returned from chief executives, 42% from clinical directors, 47% from medical directors and 59% from directors of clinical governance. Data were returned from 69 of the 79 trusts sampled (87%), ranging from 0% to 100% of forms returned (mean 48%).

Box 1. The National Institute for Health and Clinical Excellence (NICE) six principles of implementation

1. 1. Board support and clear leadership

2. 2. Provision of a dedicated resource (a NICE manager)

3. 3. Support from a multidisciplinary implementation team

4. 4. A systematic approach to financial planning

5. 5. A systematic approach to implementing guidance

6. 6. A process to evaluate uptake and feedback

Descriptive results

Corporate and local leadership (mapped to NICE principle 1)

1. • 56% indicated that the trust had identified a clinical director-level lead for guideline implementation

2. • 46% indicated that the trust had identified a clinical guideline implementation champion; of the clinical champions, 57% had a team to support them

3. • 85% indicated that the trust had a committee to oversee implementation of guidelines

4. • 57% indicated that trust boards routinely received reports on guideline implementation.

Staffing and resources (mapped to NICE principles 2 and 3)

1. • 32% indicated that the trust had appointed staff specifically to implement NICE guidelines

2. • 42% indicated the trust had calculated likely resource implications for guidelines

3. • 38% indicated the trust had committed funds to implementation

4. • funds committed ranged from £10 000 to £1.2 million (mean £34 000).

Commissioner resourcing (mapped to NICE principle 4)

1. • 84% indicated that recent NHS funding problems had prevented (or would be likely to prevent) implementation of guidelines

2. • 71% rated the trust's support from their commissioning primary care trust in implementing health technology appraisals as poor, very poor or non-existent

3. • 82% rated the trust's support from their commissioning primary care trust in implementing guidelines as poor, very poor or non-existent.

Accessibility of guidelines (mapped to NICE principle 5)

1. • 70% indicated that NICE guidelines were easily available on the trust intranet

2. • 67% indicated that the trust also made guidelines accessible in the course of clinical practice in another way

3. • 86% had looked for implementation information on the NICE website.

Operational implementation (mapped to NICE principle 5)

The use of outcome measures is a proxy for implementation, because it is a precursor for appropriate recording of patient information and an indicator that diagnosis is being recorded. Respondents were asked if clinical teams routinely used outcome measures. Responses are shown in Fig. 1. Seventy per cent of respondents indicated that their trust used a computerised clinical information system for patient details.

Monitoring (mapped to NICE principle 6)

1. • 73% indicated that the trust had audited health technology appraisal implementation

2. • 84% indicated that the trust had audited guideline implementation.

Fig. 1. Percentage of respondents indicating that clinical teams routinely use outcome measures.

Inferential results

Commissioner support of health technology appraisal implementation

One-way analysis of variance showed that a lower commissioning score is associated with:

1. • lower likelihood that a trust would have calculated the resource implications of implementation (F=2.464, d.f.=4, P=0.047)

2. • lower likelihood that resources had been committed (F=4.217, d.f.=4, P=0.003).

Correlation showed that a lower score for commissioning support is linked to a lower level of commitment and leadership (see below; R=0.180, P=0.048).

Commissioner support of guideline implementation

Our hypothesis was that, as with support for health technology appraisals, commissioning support of guideline implementation might exert an effect on other variables. One-way ANOVA showed that poorer commissioning support of guidelines was associated with:

1. • higher level of health technology appraisal audit (F=2.471, d.f.=4, P=0.046)

2. • poorer access to guidelines other than on the intranet (F=3.184, d.f.=4, P=0.005).

NHS funding problems

Our hypothesis was that funding problems might curtail implementation. This proved not to be the case for audit of health technology appraisals, where reported funding problems were associated with:

1. • higher likelihood of audit taking place (F=14.667, d.f.=1, P < 0.001);

2. • poorer level of access to guidelines other than on the trust intranet (F=7.375, d.f.=1, P=0.007).

Commitment and leadership

One-way ANOVA showed that better levels of commitment and leadership were associated with:

1. • staff appointed to support implementation (F=2.987, d.f.=12, P=0.001)

2. • resource implications calculated (F=2.153, d.f.=12, P=0.016)

3. • resources committed (F=3.025, d.f.=12, P=0.001)

4. • audit of health technology appraisals undertaken (F=2.702, d.f.=11, P=0.003)

5. • existence of a clinical information system (F=1.940, d.f.=12, P=0.032).

Correlation showed that better commitment and leadership is associated with more prevalent use of outcome measures (R=0.194, P=0.008).

Commissioner support, and corporate commitment and leadership

Correlation shows a positive relationship between commissioner support for health technology appraisals and commitment and leadership (R=0.180, n=181, P=0.016).

Analyses of the Healthcare Commission audit

A series of analyses were carried out to look at the relationship between previously identified ‘causal’ factors and implementation as measured by the Healthcare Commission audit of implementation of the schizophrenia guideline.

Descriptive results

Items within the audit were collapsed into a number of key markers showing adherence levels (Table 1).

Table 1. Adherence with key markers from the Healthcare Commission audit

	Compliance, %
Marker	Minimum	Maximum	Mean (s.d.)
Depot used where preferred or adherence issue	82.00	100.00	96.99 (4.10)
Responses/side-effects documented in care plan	60.0	100.0	96.10 (5.71)
Polypharmacy not given	67.0	99.0	92.20 (4.64)
Care plan has advance directive or continuity plan	49.0	100.0	81.60 (10.63)
Physical health review	9.0	97.9	80.98 (11.15)
Written material offered	48.0	92.0	67.81 (8.66)
Occupational status assessed	49.0	99.0	64.82 (11.46)
Clozapine prescribed for treatment-resistant schizophrenia	30.8	90.3	61.28 (11.90)
Family therapy offered	7.1	93.2	55.17 (17.30)
Cognitive—behavioural therapy offered	20.0	96.0	46.86 (15.08)
Care plan has advance directive or continuity plan and choice of antipsychotic	0	83.1	29.27 (13.10)

Inferential results

In order to look at statistical relationships, the individual scores for the audit were combined to give an overall proxy measure for implementation. Correlations showed that better levels of implementation were associated with the following factors (data at trust level):

1. • better corporate commitment and leadership (R=0.342, n=57, P=0.009)

2. • existence of a committee to oversee implementation (R=0.429, n=57, P=0.001)

3. • commissioner support for health technology appraisal (R=0.459, n=57, P=0.000)

4. • less likelihood of guideline audit having been carried out (R=–0.270, n=57, P=0.042).

The data were submitted to a step-wise linear regression. All variables from the NCCMH survey were input as independent variables, the schizophrenia proxy as the dependent. The model yielded a reasonable R² _adj of 0.284 using three variables, model ANOVA significant (F=8.410, d.f.=3, P < 0.001), and collinearity statistics were within acceptable range (condition index=7.72). Variable (β) weights showing direction and strength of effect in the model were:

1. • commissioner support for TA, β=0.39

2. • corporate support, β=0.26

3. • identified clinical director lead, β=0.23.

This interaction between commissioner support and corporate support has a powerful predictive effect on implementation.

Discussion

Limitations

The response rate was below 50%, and thus the findings must be seen as illustrative rather than definitive. To present the results from the survey we used all individual responses, as opposed to overall trust responses. There is, in some cases, variation between responses from different individuals from the same trust (although analysis of variance showed no significant results). Since the study focused only on mental health trusts, findings are not generalisable to other healthcare settings; the survey also provides a snapshot only of the current situation. The use of a self-report method (for both the questionnaire and the schizophrenia audit) will inevitably bring in elements of bias.

NICE guidance implementation and key implementation indicators

The NICE principles of implementation (Box 1) are not being closely followed by many trusts. It is encouraging, however, that a majority of trusts seem to have some identified prerequisites for implementation in place, showing engagement with guideline implementation, although it is arguable that these might be seen as prerequisites for an effective clinical service, and that their existence owes more to this than any link to guideline implementation.

Markers from the schizophrenia audit

Some of the markers show high levels of adherence (use of depot medication, documentation of response and side-effects in care plan, avoidance of polypharmacy), whereas others are poorly adhered to (notably provision of CBT and family interventions). The high levels of adherence on the top five markers may be due to coincidence with established best practice rather than to change driven by guidance. The low levels of CBT and family interventions are possibly due to lack of availability and funding rather than commitment from clinical teams to implementation.

Key findings

Hypothesis testing shows a clear link between corporate commitment and leadership and implementation of guidelines. Corporate commitment is linked to many key implementation markers - where it is lacking, implementation will not be very far advanced. This hypothesis is clearly supported by the secondary analyses of the Healthcare Commission data on NICE schizophrenia guideline implementation. There is a clear link between corporate commitment and leadership and implementation of this specific guideline, from both the correlation and the regression analyses, the latter showing the interactive effect of commissioner support and commitment and leadership (lack of commissioner support is linked to poorer corporate commitment and leadership).

It is interesting to note that a higher level of audit of health technology appraisals is linked to poor commissioning support and funding problems. This may be a function of two possibly unrelated effects: it may be that commissioners are insisting that an audit of an appraisal is undertaken as a prerequisite of funding being allocated; it is also possible that this audit is being conducted as a lobbying activity to persuade commissioners to allocate funding (via a demonstration of need).

Implications

This study shows empirically that in order for implementation to succeed, the trust's board and senior managers must be engaged. Further, it shows that commissioners must demonstrate support. These findings give ammunition to trusts and to NICE to enable pressure to be put on commissioners in order to improve support of NICE guidance and of its implementation. Further, these data show that some trusts have advanced well in implementing guidelines, providing identifiable best-practice models for others to follow.

Declaration of interest

None. The study was funded by the National Collaborating Centre for Mental Health.