Collecting service use data for economic evaluation in DSPD populations

Abstract

Background

Economic evaluation of the Dangerous and Severe Personality Disorder Programme is essential to ensure value for money. The collection of individual-level service use information is crucial to any such evaluation, but the best way to collect these data in secure facilities is unclear.

Aims

To develop a method for the collection of individual-level service use information for prisoners/patients in secure facilities.

Methods

Services provided within secure facilities were identified through examination of facility and policy literature, and discussions with managerial and clinical staff. Appropriate methods of measuring the quantities of services used were then explored and a new research tool capable of capturing all relevant services was developed and pilot tested.

Results

The Secure Facilities Service Use Schedule (SF–SUS) records service use information from records and is capable of capturing data on the use of all individual-level services provided within a secure facility plus external services commonly accessed by occupants.

Discussion

The SF–SUS is able to collect meaningful individual-level service use information for the economic evaluation of services provided within secure facilities.

Research in personality disorders has long been neglected and there is little evidence of the clinical effectiveness (Warren et al, 2003) or cost-effectiveness (Swaray et al, 2005) of alternative interventions. The care of people with severe personality disorders, such as dangerous and severe personality disorder (DSPD), is now receiving overdue attention. Substantial funding has been devoted to a programme of new assessment and treatment services for this group and the DSPD Programme is subject to rigorous clinical and criminal evaluation (see Hart et al, 2007; Tyrer et al, 2007, this issue). Scrutiny from an economic perspective is also important because the DSPD Programme needs to be justified in terms of cost-effectiveness as well as effectiveness. Economic evaluation is the comparative analysis of alternative courses of action in terms of both their costs and consequences and is an established practice in the appraisal of health services (Drummond et al, 2005) and mental health services (Knapp, 1995).

A current evaluation of the assessment of DSPD in secure facilities (the IMPALOX study, details available from authors), included an economic component, and provided an opportunity to explore the methodological implications of undertaking an economic evaluation within a secure facility.

Typically, the cost of supporting patients/prisoners in secure facilities is calculated using a top-down approach. Top-down costing involves calculating the total cost of all resources and dividing them by the total population of the secure facility. However, economic evaluation requires individual-level cost data, rather than average costs that do not vary according to prisoner/patient (Drummond et al, 2005). To calculate individual-level costs, individual resource use must first be identified and measured (Beecham & Knapp, 2001). Identification involves drawing up a list of all resources relevant to a particular prisoner/patient. Measurement is the means by which data on the quantity of resources are collected. One obstacle to the successful completion of the IMPALOX economic evaluation was the lack of established methods of measuring resource use in secure settings. This paper explores the characteristics of people referred for DSPD assessment and/or treatment and the nature of secure facilities, in an attempt to develop a method of measuring service use information that is appropriate for research into the economics of secure facilities.

METHOD

Identification of resources

The first step in the calculation of costs for the purpose of undertaking an economic evaluation is the identification of resources. Resources relevant to a DSPD population were identified through a review of relevant literature and policy material. Information on interventions offered as part of the DSPD assessment and treatment programme were collected from a number of policy and service-level documents. (Home Office& Department of Health, 1999; Home Affairs Select Committee, 2000; National Institute for Mental Health in England, 2003; DSPD Programme, 2005; Hart et al, 2007; Tyrer et al, 2007, this issue). General prison and secure hospital services available to all prisoners/patients in secure facilities were located through a review of annual reports of prisons and secure hospitals. In addition, managerial and clinical staff at one prison providing a DSPD assessment and treatment programme and one secure hospital were asked for comments on the services identified through the literature.

Measurement of resources

Once identified, there are a number of methods available to measure the quantity of resources used by participants in an economic evaluation, including questionnaires, diaries or searches of case notes (Byford et al, 2003). In healthcare evaluations, resource use is commonly measured in interviews with patients by using a service use schedule such as the Client Service Receipt Inventory (CSRI; Beecham &Knapp, 2001). Although service use questionnaires often need to be adapted for different participant populations, the principle disadvantage is the need to rely on the memory of interviewees over what can be a significant number of months. Service use diaries are one method of improving recall and involve asking participants to record their use of services prospectively over the study period. Diaries can be highly structured and involve simple tick boxes. However, the more complex and broad the range of services used, the harder it becomes for diaries to remain manageable. An alternative method is to collect retrospective information from case notes or electronic administrative databases. Records are likely to be more accurate than relying on user recall over a substantial period of time, but record searches can be time consuming, may not record exactly the information needed and will often be hampered by poor completion, missing files and illegible entries.

There are three areas where the DSPD Programme challenges the application of existing methods for collecting service use information. First, DSPD is a complex condition. The Programme involves complex interventions with multiple goals, multiple agencies and a high degree of user involvement (Campbell et al, 2000). Individuals meeting the criteria for DSPD are a heterogeneous group and outcomes are multiple. The complexity of the DSPD intervention has clear implications for the measurement of resource use: individuals are likely to see a wide range of professionals, so data on a broad range of resource use must be collected, making diaries, in particular, a complex and significant burden on respondents.

Second, there may be problems in completing research in individuals with DSPD. By definition, prisoners/patients in the DSPD Programme are challenging individuals. The development of clinical diagnostic interviews for personality disorder has highlighted the difficulties of eliciting accurate responses in interviews (Tyrer & Ferguson, 2000). Therefore collecting service use information directly from prisoners/patients, whether through interview or diaries, might not be appropriate.

Third, there are practical difficulties in carrying out research in secure facilities. For example, access to secure facilities and to prisoners/patients is strictly controlled and requires appropriate permission and/or security clearance. The research must be undertaken alongside the regime of prison/secure hospital, making access to participants more difficult than community-based research, particularly where access is required for long periods of time. Interviews may be interrupted and agreed access may be denied as a result of security alerts.

Despite these difficulties, secure facilities offer one significant advantage for the collection of individual-level service use data compared with community-based research. Resources available in institutional facilities are essentially limited to the services provided on site and although records held in each institution will not record all resource use, they should be sufficient to capture the majority of the services available to prisoners/patients. Although some services are provided by external agencies, the secure nature of the facilities means that such contacts are closely monitored and recorded. Thus, data collection from records is likely to be the most appropriate method of measuring service use in secure facilities.

To ensure that data are collected systematically, a service use schedule is needed, covering the wide and varied range of services provided in secure facilities. None currently exists for the collection of data from records, rather than prisoner/patient interview. We therefore developed a new research tool for the measurement of resource use in secure facilities.

RESULTS

Secure Facilities Service Use Schedule

The Secure Facilities Service Use Schedule (SF-SUS) was based originally on service use schedules for economic evaluations designed by S.B. - the AD-SUS for adults and the CA-SUS for children and adolescents (see Byford et al, 1999; Barrett et al, 2006) - but was substantially modified on the basis of information collected in the identification phase of the research. Services were separated into three sections: accommodation, service use within the secure facility and service use outside the secure facility. A draft schedule was developed and piloted on 16 patients with personality disorder in a secure hospital and two prisoners in a high-security prison. All participants were being considered for the DSPD Programme.

The pilot study uncovered a number of issues concerning both the type of data and the way in which they are collected. In particular, services that were unique to secure settings were identified, such as travel between institutions requiring security escorts and drivers and non-face-to-face contacts with legal representatives (telephone calls and letters). These services were added to the SF-SUS. The pilot study also enabled us to include suggested data sources in the schedule. We were also able to identify information available electronically and that kept in paper files, and the likely difficulties in gaining access to different data sources. The revised SF-SUS is shown in the data supplement to the online version of this paper. The schedule takes around 2 h to complete, although this varies according to the availability of patient records. Information is recorded for a specific determined period (e.g. 6 months).

Accommodation

The accommodation section of the SF-SUS asks for the name of the institution, the wing or the ward and the number of days spent in each location. In prisons, this information can be gathered from the computer database ‘Inmate Information System’ or from the wing record. In secure hospitals, the information can be found in the patient's medical files. For those in prison, the accommodation section also asks whether the prisoner was subject to any special measures such as a Rule 45 order (segregation) or an open F2052SH form (at risk of self-harm), since such measures result in more intensive staffing and therefore higher costs. Information on special measures is kept in a wing record or inmate personal record. Movement between secure facilities and the number of escorts involved are also recorded in the accommodation section.

Service use within secure facilities

Information on service use within secure facilities is held in a number of different locations. In prisons, inmate medical records and wing records provide information on contacts with health and social care professionals and periods spent in the prison hospital. For those in other secure facilities, this information should be available in personal medical records. The SF-SUS gives sections for contacts with 12 categories of professionals and asks for the number of contacts and the average duration of each contact within the specified study period. There is also space to enter the details of contacts with professionals not included in the list. As well as individual contacts with health and social care professionals, prisoners/patients can take part in a range of daily activities. The SF-SUS records the number of hours spent in different types of activities, including therapeutic groups, educational courses, creative activities, work and sports activities and games, each of which incur a cost. Finally, information is collected on the number of complaints that a prisoner/patient makes during the study period, as some complaints can result in the involvement of the governor and occasionally, external organisations.

Services external to secure facilities

Prisoners/patients have access to a range of services external to the institution, particularly those provided by the health and criminal justice sectors. Healthcare received externally in local National Health Service hospitals is recorded in prison/secure hospital medical files and the SF-SUS records the name of the hospital, the type and number of contacts, the length of stay if appropriate and the medical specialty. Contacts with professionals from the criminal justice system, including police officers and legal professionals, are recorded in prisoner's wing records or secure hospital patient files. Telephone calls and letters in prison are logged, allowing telephone contact with legal professionals and letters from legal professionals to be traced and counted. In addition, parole board hearings and mental health review tribunals are recorded on prisoner wing records and in secure hospital patient files.

Other resources

The SF-SUS elicits information on the use of services that can easily be identified, measured and recorded on an individual basis. There are, however, many other resources that cannot be directly allocated on an individual basis, for example security staff, utilities and administration. These resources should be treated as overheads and incorporated using a top-down approach (total cost of all overheads divided by the total population of the secure facility). Overhead costs are then added to the individual level costs, calculated on the basis of data collected with the SF-SUS.

DISCUSSION

Economic evaluation of the new DSPD services is crucial to ensure value for money, but existing research methods are not necessarily suitable for use in secure settings. In particular, we identified the lack of established methods for measuring resource use in secure settings. To develop an appropriate research tool to measure resource use for the purpose of economic evaluation, the characteristics of people referred for DSPD assessment and/or treatment and the nature of secure facilities were explored. Prison and secure hospital resources relevant to a population with DSPD were identified and a service use schedule, the SF-SUS, was designed and tested. The SF-SUS obtains service use information from records, rather than interviews, and covers all services relevant to occupants of secure facilities.

The SF-SUS has been used successfully to collect and cost service use information in a secure hospital in a sample of patients being considered for DSPD assessment (Barrett et al, 2005). The average cost per participant was estimated to be£65 545 over 6 months, ranging from £59 119 to £82 709, which indicates that despite the necessity of allocating a large proportion of overhead costs uniformly, the SF-SUS is able to identify substantial variations in service use and cost between individuals. The SF-SUS is currently being used in a number of economic evaluations in secure facilities.

There are a number of problems with the approach to collecting and costing service use information outlined here. First, collecting individual-level service use data in secure facilities is time consuming and can be frustrating. The ease with which data are collected depends upon gaining appropriate permissions and security clearance, the cooperation of the staff and access to all relevant prisoner/patient records. Even with full access, the researcher must abide by the regulations of the secure facilities and may find access denied at times of heightened security. These practical difficulties must be taken into consideration at the planning and funding stages.

Second, detailed costing of individual items of a complex service can never be completely accurate. The researcher is reliant on the accuracy of the records from which the data are collected and the accuracy of the unit costs applied to each item of service. Individual-level costing tends to produce lower estimates than all-inclusive top-down approaches, which involve dividing the total cost of a secure facility by the population of that facility. However, the average individual-level cost calculated in our pilot study was within £1000 of the top-down estimate, suggesting a relatively high degree of accuracy (Barrett et al, 2005). The secure nature of these facilities and the close monitoring of individuals help to produce a level of accuracy that is less likely in community-based research.

Prisoners/patients taking part in the DSPD assessment and treatment programme can have access to all the services and activities provided within secure facilities, and the SF-SUS is designed to capture this full range of resources. The SF-SUS is therefore an appropriate research tool for the collection of service use data for all residents of secure facilities, not just those within the DSPD Programme. Although some categories of prisoners/patients, such as those currently considered appropriate for the DSPD Programme, might be unlikely to return to the community during the course of a typical research project, this might not be the case for all prisoners/patients. For populations in which movement between secure facilities and the community is anticipated, a second version of the SF-SUS is available. This is capable of capturing services in either location.