User-led assessment of a recovery service using DREEM

Abstract

Aims and Method

To develop a service development plan supporting recovery-based practice through collaborative service user-led methodology, and contribute to the National Institute for Mental Health in England (NIMHE) pilot evaluation of recovery sensitive measures. Staff, residents and service user group representatives collaborated in implementing the Developing Recovery Enhancing Environments Measure (DREEM) in a residential rehabilitation service.

Results

Staff and residents demonstrated a shared vision of recovery. Many areas were being addressed to a high level but others showed areas of deficiency, and illustrated discrepant views between staff and residents, which provided a focus for the development of a clinical service action plan.

Clinical Implications

As an experimental measure DREEM provides a user-led structure, which enables services to measure their commitment to, and effectiveness in, providing recovery-based care. It focuses service development and enhances collaborative work with service users, thus mirroring the principles of recovery it measures. It merits further evaluation and consolidation.

The concept of recovery in mental health services has gained increasing prominence over recent years, but can be traced back 200 years to the Tukes at York (Roberts & Wolfsen, 2006), who established a clinical philosophy based on kindness, compassion and hope of recovery. It has perhaps been best defined by Anthony (1993) who described personal recovery as able to occur in the presence of ongoing symptoms but involving ‘ a way of living a satisfying, hopeful and contributing life even with limitations caused by illness’. These principles have been adopted by many modern mental health services, including those in New Zealand (Mental Health Commission 1998, 2005) and the United States (President's New Freedom Commission on Mental Health, 2003). In the UK both the National Service Framework for Mental Health (Department of Health, 1999) and the NHS Plan (2000), summarised as The Journey to Recovery (2001), are built upon an aspiration to develop recovery-based practice, practitioners and services. Most recently ‘ promoting recovery’ has been identified as one of the ‘ essential shared capabilities’ (Hope, 2005) in which all mental health staff, including psychiatrists, should be skilled.

Mental health services in Devon, including the Exeter and East Devon Rehabilitation and Recovery Service, have been identified as a ‘recovery development site’ by the National Institute for Mental Health in England (NIMHE). This includes both residential and community services, with an overarching goal of supporting pathways to recovery, in line with contemporary developments in rehabilitation psychiatry (Roberts et al, 2006). Although clinical practice has been supported by adoption of a recovery philosophy, there is a need to produce simple, meaningful and robust measures of both individual and service progress to enable evaluation and guide development. There are no fully developed tools with which to evaluate the recovery orientation of a service, but NIMHE has identified the Developing Recovery Enhancing Environments Measure (DREEM) as the most promising of an emerging group of recovery sensitive measures (Campbell-Orde et al, 2005) and produced a preliminary UK version (information available from [email protected]).

Working with the national NIMHE recovery leads we undertook a pilot study, using DREEM, as a stimulus to our local service development and a contribution to building a case for a national study. We sought to conduct this evaluation in a ‘recovery-based’ manner, which we interpreted as devising a collaborative user-led methodology in which there was joint ownership of the project, underpinned by formal guidelines recently published on the ethics of user-led research (Faulkner, 2004). We also aimed to acknowledge and respect the differing viewpoints of staff and patients, and construct an action plan for service development based on an awareness of these similarities and differences.

The DREEM is a substantial questionnaire in seven sections yielding 160 data-sets. An early decision of the advisory group set up to steer the study was to select three elements to trial, which included demographic, quantitative and qualitative measures. These include self-reported ratings of the importance of components of recovery and how well services address these. Each component is broken down into sub-components describing the element in everyday language. This paper reports the process and outcomes of the study but confines itself to data from the quantitative evaluation of the perceived importance of recovery factors and how well service users and staff believed that the service addresses these factors.

Method

An advisory group was established comprising ward residents, service user group representatives, medical and nursing staff. All decisions were made collaboratively within this group ensuring joint ownership of the purpose, process, interpretation, outcomes and reporting of the study.

The service user group representatives initially met with the residents on our in-patient residential unit to explain the study and gain their consent. The representatives acted as user-researchers, supporting residents in completing DREEM. Staff from the same unit were asked to complete the questionnaire during a staff and service development day. Their responses were presented back to them on the day and compared with residents’ responses (Figs 1 and 2). Discussion and reflection upon these results, with further support from service user group representatives, led to an initial action plan. Following this, a consultation period with residents and the representatives was used to develop the initial plan into a formal action plan for developing our residential recovery-based practice over the next year, including a commitment to repeat the study to monitor progress.

Statistical analysis was performed using SPSS version 11.1 for Windows. Two-tailed t-tests were performed to assess the significance of the difference between responses by staff and residents.

Fig. 1. Staff and resident ratings of importance of factors.

Results

Out of 14 residents, 10 chose to complete the questionnaire. The 4 residents who declined each gave significant reasons, which are included with the qualitative responses in the full report (available at http://www.recoverydevon.co.uk). Most residents were male (n=9) and White British (n=9), covering a broad age range (range 22-60 years). They primarily had long-standing diagnoses of psychotic illnesses and prolonged involvement with services.

The questionnaire was completed by all 26 staff. Staff were more equally matched in gender; male (n=11) and were older (46% over 45 years). They were also predominantly White British (n=21) and have significant experience of working in mental health services (14 staff have over 10 years’ experience).

In the first part of the analysis staff and residents rated the 24 components of recovery according to their importance (Fig. 1). Components were rated from 2 (strongly agree that factor is important in recovery) to –2 (strongly disagree that factor is important). The components were then rated according to how well staff and residents believed each factor was being effectively addressed by the service (Fig. 2). Components were again rated from 2 (strongly agree factor is effectively addressed) to –2 (strongly disagree factor is effectively addressed). A more full description of components is given in Table 1.

Table 1. DREEM on Russell Clinic. How well factors of recovery are achieved

Factor description	Residents’ rating of achievement (mean)	Staff rating of achievement (mean)	Staff — residents mean difference	P 1
Self-identity	0.8	1.3	0.5	0.266
Meaning in life	0.6	1.1	0.5	0.304
Hope	0.7	0.8	0.1	0.339
Knowledge of illness	0.4	0.9	0.5	0.108
Self-management	0.6	1.0	0.4	0.133
General health	0.6	1.1	0.5	0.087
Active in own recovery	0.3	0.7	0.4	0.079
Personal rights	0.4	1.1	0.7	0.002**
Peer support	0.6	1.0	0.4	0.149
Meaningful activities	1.1	0.9	-0.2	0.565
Community	0.3	1.0	0.7	0.047*
Personal relationships	0.3	0.8	0.5	0.042*
Build on own strengths	0.5	1.0	0.5	0.109
Developing new skills	0.6	1.0	0.4	0.205
Basic needs met	0.4	0.7	0.3	0.353
Sense of control in life	0.7	0.8	0.1	0.802
Spirituality	-0.7	0.7	1.4	< 0.001***
Social roles	-0.2	0.6	0.8	0.001***
Challenging stigma	-0.2	0.9	1.1	< 0.001***
Facing new challenges	0.6	0.8	0.2	0.137
Role models	-0.2	0.6	0.8	0.011*
Help in crisis	1.1	1.0	-0.1	0.821
Sexuality	-0.5	0.5	1.0	< 0.001***
Helpers who care	0.6	1.1	0.5	0.059

There was no statistically significant difference between how staff and residents rated the importance of each factor. However, there was significant discrepancy between staff and residents in how they rated the effectiveness of the service in addressing each factor (Table 1).

Discussion

The contents of DREEM items were derived from service users’ views on what was important to them in supporting their recovery. The finding that staff and residents strongly agreed with this profile can be taken as a positive affirmation that the team and service are, to at least some extent, achieving the aim of adopting a recovery-based clinical philosophy. This indicates a compatible vision towards recovery being shared by staff and the service users with whom they work.

Fig. 2. Staff and resident ratings of level of achievement of factor. *P<0.05, **P<0.01, ***P<0.001.

There was also general agreement between staff and residents on the relative degree to which these factors were being supported in the service, although significant differences (P≤0.01) with respect to sexuality, spirituality, social roles, challenging stigma and personal rights and a general trend for staff to rate the service effectiveness higher than the residents was noted.

Working with the service user group, the service development plan was constructed to give particular attention to those factors that were either perceived as being poorly addressed or where staff and residents had discrepant views. Those items that were negatively scored by residents were also the factors with the greatest level of difference between staff and resident ratings. This demonstrates the value of a tool that gains responses from multiple sources and enables a comparison of viewpoints.

There are significant limitations to the conclusions that can be drawn from this evaluation. The DREEM itself has not undergone a full psychometric evaluation and so aspects of reliability and validity are unknown, and the results must be interpreted in this light. For instance, the impact of daily fluctuations in service user attitudes towards services, or variance in user evaluation at different stages of their recovery is unknown. The service user group compiled a report on weaknesses in the application of DREEM during the course of the study. These included confusion over the meaning of some items which was fed back to national NIMHE recovery leads. Obvious demographic differences between the staff and resident groups may also have had an impact on the results. This does not reduce the significance of these findings however, but simply offers a possible explanation for their source.

Other limitations include the small numbers included in this study which leads to the increased possibility of type II errors. This issue can only be addressed by a larger scale study. The large amount of data and the subsequent large number of t-tests performed increases the risk of type I errors. Focusing only on those factors found to be significant at 1% rather than 5% reduces this risk.

The team valued DREEM as providing a clear and structured model of recovery for staff and residents to use in assessing their service. In this way, the DREEM survey had a significant educational component and promoted both reflective and collaborative practice with service users, that was valued by staff and residents. The aspiration to use recovery-based, user-led methodology in conducting the study ensured that service users and representatives were involved at all stages of the process and included iterative techniques to ensure constant consultation with service users, leading to co-ownership of the project. This is also likely to have assisted the team in achieving the high response rate in this project. In particular the service users made it clear from the beginning that their cooperation was dependent upon the project making a difference to the service they received. This was achieved by coupling the DREEM survey to development of the annual service action plan. This led to a series of practical decisions, including increasing one-to-one time between residents and staff, implementing a routine physical health screening check-up, new information packages being developed, the advocacy service being reviewed and the chaplaincy service being enhanced. All actions are designed to address specific areas of concern highlighted by this project.

This has been an interesting experiment that staff and patients have responded to with enthusiasm, and was experienced as a further step towards listening more clearly to the user voice and working more collaboratively to develop the services that people need, want and value. Although there is acceptance of the move towards user-centeredness at the highest levels of our profession, there remains considerable difficulty in knowing how to operationalise this radical aspiration, and move beyond mere ‘ representation’ in practice. DREEM presents an opportunity of meaningful collaborative evaluation and planning with service users in developing recovery-based practice. Overall, it has proven to be an effective and useful device with which to appraise and develop our service.

From our experience we would recommend DREEM to other services wishing to reflect on their effectiveness in providing recovery-based care and also support NIMHE in their commitment to establishing a larger-scale study to further evaluate and refine the DREEM as a step towards more meaningful measurement of recovery process and outcomes.

Declaration of interest

None.