Setting up clinical audit in a psychodynamic psychotherapy service: a pilot study

Abstract

Aims and Method

Clinical governance implies a need to engage in a demonstrable form of clinical audit. We decided to pilot a pre–post-therapy questionnaire study, involving both therapists and patients, with the aim of assessing its feasibility as a routine measure of outcome in our service. Questionnaires were chosen to reflect both the symptom profile Clinical Outcomes in Routine Evaluation (CORE) and the general level of functioning Global Assessment of Functioning Scale (GAF). The patients were also questioned about their satisfaction with the therapy.

Results

Of 53 eligible patient–therapist pairs, 26 patients and 19 therapists responded pre-treatment (overall 51% response rate). The mean (s.d.) CORE score per item was 1.93 (0.78), whereas the GAF score was 55 (15.2), somewhat belying the idea that psychotherapists only treat the ‘worried well’. The post-therapy response rate was poorer, rendering statistical analysis of change difficult to perform. The trend, however, was towards an improvement in both symptoms and level of functioning over the course of therapy.

Clinical Implications

Our patient group presents with a considerable degree of impairment. Although the response rates in this pilot study were poor, the data trend suggests that patients do benefit from the treatment offered. The aims of the study were met in that implementation of a clinical audit process within our department is feasible and the data are obtainable. The main interest, however, lies in what we learned from the process. We offer a dynamic and practical assessment of the impediments encountered, with implications for how these might be confronted.

With the increasing emphasis on clinical governance in health care provision, all NHS services are finding themselves in the position of having to justify their work using standardised outcome measures. Clinical governance can be defined as ‘the means by which healthcare organisations ensure the provision of quality care by making individuals accountable for the setting, maintaining and monitoring performance standards’ (Lawrie et al, 2000). One of the key elements of this is the clinical audit loop. At a local level this can provide essential data on performance and identify areas where improvements can be made.

Traditionally, the psychodynamic psychotherapies have been criticised for their lack of evidence base and the services for failing to promote a culture of systematic evaluation, including that of clinical audit. This could be due in part to the reluctance of therapists to be involved in a process that could potentially affect the transference relationship with their patients. In addition, the outcome of such therapies is relatively difficult to measure. However, if such services are to continue to operate and receive funding in the modern culture of the National Health Service (NHS), which emphasises accountability and transparency, they will have to develop such systems of evaluation to justify their practice.

Our unit had been practising a form of clinical audit using the Kaizen approach for many years (Feldman & Pugh, 1998), and found it an effective means of monitoring clinical practice. However, these data were largely qualitative and the need to supplement them became increasingly apparent, particularly with respect to outcome. In line with this trend, the Association of Psychoanalytic Psychotherapists (APP) instituted a multi-centre naturalistic outcome study of psychotherapy in the NHS in the late 1990s. This involved several questionnaires at multiple time points in the therapy. Having piloted its use in our unit, we found it too cumbersome to administer, given our resources at that time.

A review of the literature has revealed a lack of research into the implementation of a manageable clinical audit suited to a modern, busy NHS psychotherapy department such as ours. The aim of this project was to set up a cycle of audit to measure outcome in a form that was feasible in our unit, with no additional resources. We were interested both in changes in patients' symptoms and in their satisfaction with our service, the latter not being emphasised by the APP study. The initial cycle, reported here, was a pilot study to assess the practicability of such an audit and to identify early problems. Interestingly, the project proved more difficult than anticipated, and this paper is an attempt to evaluate some of the problems experienced, as well as to report our preliminary findings.

Method

The project was conducted between November 2000 and April 2001. We aimed to include all patients who were offered individual psychodynamic psychotherapy in our psychotherapy unit at the Maudsley Hospital during that time. Group, family and cognitive-behavioural therapy (CBT) patients were excluded from the initial cycle (pilot study).

Reliable, validated rating scales were chosen as measures of psychological symptoms and levels of functioning, as well as a more descriptive questionnaire to assess patient satisfaction. The tools used were the Global Assessment of Functioning Scale (GAF, total score; American Psychiatric Association, 1987), completed by the assessor, the Clinical Outcomes in Routine Evaluation (CORE; Evans et al, 2000, 2002) questionnaire, and the Client Satisfaction Questionnaire, and Patient Questionnaire (CSQ/Q-P) completed by the patient.

The GAF is the assessor's assessment of the patient's overall level of functioning (American Psychiatric Association, 1987). It can be divided into two sub-scales, one for symptoms and one for disability. For the purpose of this study, a total score was used. The scale consists of a hierarchy of statements describing the subject's current psychological, social and occupational functioning, labelled from 0 (poor functioning/severe symptoms) to 90 (high functioning/few symptoms). The clinician estimates the level of functioning by means of a single value, guided by the descriptive statements. Broadly, scores of 1-40 can be equated with severe functional impairment needing community mental health team input, those of 41-70 with moderate to severe impairment warranting local psychological therapies services and those of 71-90 as warranting primary care (NHS Executive, 2002).

The CORE is an acceptable, standardised outcome measure designed to assess efficacy and effectiveness across multiple disciplines offering psychological therapies (Evans et al, 2000, 2002). It consists of 34 statements referring to the subject's state of mind over the preceding week. Each statement is scored from 0 (symptom absent) to 4 (symptom severe), the final result usually presented as an average per item (0-4). The score can be subdivided into those for well-being, functioning, risk and other problems. We used the global score only. Normative data collected from clinical and non-clinical populations have provided mean (s.d.) CORE values of 1.86 (0.75) and 0.76 (0.59), respectively (Evans et al, 2002).

The CSQ and Q-P were chosen to assess patient satisfaction with the service they received and with the therapeutic process, respectively. The CSQ is a general measure of satisfaction with services, which has a high degree of internal consistency and correlates well with therapists' estimates of client satisfaction (Larsen et al, 1979). We used the shorter version, consisting of three questions. This was joined with Cluster 2 of the Patient Questionnaire (Q-P), which measures degree of perceived change from the patient's perspective and has been used as a stand-alone measure of perceived change in several studies (Lunnen & Ogles, 1998). Thus, the combined instrument consisted of seven structured questions that could be scored, together with an opportunity for patients to write a descriptive comment regarding their experience.

At time point 1, prior to the first session, each patient was sent a copy of the CORE by post with a stamped addressed envelope. The therapist was asked to complete the initial GAF following the session. This was designed to establish both objective and subjective baseline measures of symptomatology while minimising interference with the emerging patient-therapist relationship.

At time point 2, during the penultimate or last session of therapy, the patient was handed a second copy of the CORE by the therapist, along with the satisfaction questionnaires to complete and return by stamped addressed envelope. The therapist was asked to complete the second GAF.

In order to preserve patient confidentiality, while ensuring that data could be matched, each questionnaire was marked only with a numerical code.

Results

During the study period, 53 patients were eligible to participate. We obtained pre-treatment data from at least one of the therapist-patient pairs in 27, which is a response rate of 51%. Of those who answered, 55% were female and the mean age was 37 years (range 21-50 years). In only six subjects were all forms completed, which is 11.3% of those eligible. Of the 27 responders, we received 26 of the initial GAF forms but only nine of the final forms (therapist completed). Correspondingly, 19 of the initial CORE questionnaires were completed by patients versus eight of the final forms and only seven CSQ/Q-P questionnaires.

The overall response rate was relatively poor. The mean (s.d.; range) initial GAF score was 55.2 (15.2; 30-85), which corresponds to moderate to severe symptoms (i.e. difficulty in social or occupational functioning). The mean (s.d.; range) final GAF was 61.2 (13.1; 40-85), which is a slight improvement but not statistically significant. Likewise, the mean (s.d.; range) initial CORE score was 1.93 (0.78; 0.47-2.94), corresponding to that of a clinical population, compared with a final mean (s.d.; range) score of 1.69 (0.99; 0.26-2.70), reflecting a non-significant improvement.

Discussion

The project was more difficult to carry out than anticipated and we became interested in why that might be. In what follows, we address several aspects, from the practical and rational to the more unconscious institutional forces that may have been at work.

The limitations of the data set are evident. Our response rate was poor, with the small numbers rendering statistical analysis of differences before and after treatment fairly meaningless. However, it should be borne in mind that this was a pilot audit study. We would anticipate that in future cycles, statistical power could be increased by improving the response rate and the overall number of subjects. Similarly, in a larger study it would be important to look at clinically and statistically significant changes using the appropriate methodology (Jacobson & Traux, 1991).

Having stated the problems, it is worth commenting on the initial scores because our response rate is probably not outside the range for such studies. These scores suggest that within the group there is at least a moderately high measure of symptomatology and functional impairment, which calls into question the oft-levelled criticism that out-patient psychotherapy departments only treat the ‘worried well’. We believe that this is worth reporting, particularly as it conforms to our clinical experience. Secondly, although response at time point 2 was poor, those who did respond had shown some improvement. The main purpose of this paper, however, is not to defend the data but to consider the reasons why carrying out a relatively simple but essential study should pose problems within a psychodynamic psychotherapy unit.

The purpose of clinical audit is to examine working practice and identify areas that may be improved.

Consciously, we all applaud this. However:

'A variety of emotional pressures and anxieties give rise to taken-for-granted routines of the service and mitigate against changing these. It is the return of these feared elements in the work which constitutes the greatest threat embodied in evaluation…the element of appraisal increases this sense of threat. Any system of evaluation can all too easily come to feel like an accusation of inadequacy (Leiper, 1994: 201).

On the other hand:

'the current fashion for quality assurance can be viewed with some suspicion as an inappropriate attempt to objectify difficult choices about values and priorities, and to dispense with inevitable conflicts and uncertainties by hiding behind the appearance of scientific method (Leiper, 1994: 201).

Thus, senior staff may be uncomfortable with a process that they see to be politically necessary but of dubious clinical value, and that they know is going to add a burden to already overstretched staff. Clinical audit lends itself well to quantitative data such as length of waiting lists or type of therapy offered; it is less easy to apply to complex areas such as ‘psychic change’ or quality of relationships. However, such views are open to challenge; increasingly, the NHS requires accountability from its staff, much as insurance-funded treatments do in other countries, and complexity is no defence against the requirement for data. Last year's International Psychoanalytic Association research conference took outcome as its topic (What Works?), which is an acknowledgement of the widespread and pragmatic need for these measures. It was in this climate that the current audit was undertaken and various problems were encountered.

As alluded to above, the first and possibly the greatest hurdle to this kind of study is related to the culture of psychodynamic psychotherapy. In contrast to CBT, for example, it is not historically supported by strong experimental research such as randomised clinical trials, and this way of thinking has not been incorporated into the therapy. To illustrate this point, part of the CBT programme itself includes the completion of rating scales for monitoring patient progress, but this does not happen in psychodynamically-based treatments. Consequently, an important stumbling block was that one of our (psychodynamic) secretaries did not see it as part of her job to post the questionnaires. Asked whether she was sending out CORE questionnaires to patients at the beginning of therapy as requested, she replied rather laconically, ‘I don't know, do I?’ Rather than criticise the individual, her response could be seen as part of a culture that would tolerate such indifference.

From the therapists' viewpoint, it has been argued that their involvement, for example by prompting a patient to complete forms, may jeopardise the transference relationship and interfere with the therapeutic process. This argument seems weak, because the study has been designed in such a way that the administrators, rather than the therapists, are those seen to be conducting the trial. However, it suggests a sense of the therapy being somewhat contaminated by the introduction of these measures. In addition, the emotional burden of working with psychotherapy patients should not be underestimated. Trainees may find it hard to tolerate even staying in the same room for 50 min with a patient who stirs up feelings of helplessness, guilt, irritation, frustration, boredom and despair. Similarly, administrative staff spend a significant amount of time on the telephone talking to tearful, angry, complaining, confused or vague patients. ‘Forgetting’ to complete the additional tasks may, in part, be an unconscious enactment of ‘hate in the countertransference’ (Winnicott, 1975).

Psychotherapists in the NHS enjoy a position vis-à-vis their general psychiatric colleagues of being largely protected from the hurly burly of acute psychiatry on the one hand and having the very slow pace of change of chronically psychotic patients on the other. They may be envied and resented for being able to limit their workload and keep regular hours (i.e. for making time to think). They can then unconsciously experience the increased demands for openness and accountability as an envious attack on the privacy and autonomy of psychotherapy practice, with the result that they unconsciously resist the reasonable expectations that ordinary monitoring and measures of outcome be applied.

In recent years the several reorganisations of the NHS, with the ever-present requirement that costs be controlled and productivity increased, have added to the strain all round. Psychotherapy has been seen as a soft target for cost cutting, whether by freezing posts or abolishing whole services. Psychotherapists may be forgiven for feeling that the audit system may be advocated more with the hidden agenda of dismantling their service than of really seeking to improve it.

Menzies-Lyth, among others, has written about institutional defences against anxiety, and psychotherapists are not immune (Menzies-Lyth, 1990). This work describes the strains that health and social care organisations suffer, particularly in terms of unlimited expectations of the type fuelled by politicians who promise more services for lower taxes. However understandable, resistance to change as a collective response to the anxiety mobilised is not ultimately an option.

Finally, other hurdles to the completion of this study were purely practical in nature. Keeping track of the large number of part-time and honorary therapists attached to the department was particularly troublesome. In addition, many of these staff are ‘voluntary’, seeing patients out of hours, unpaid, so that the imposition of additional tasks was more difficult. Other problems are those probably encountered by most researchers, including forms being lost, unlabelled or incomplete, the difficulty of tracing those who drop out of therapy early and the various biases intrinsic to questionnaire-based research. Many of the problems encountered may not be specific to our unit. As Isaac Marks has pointed out from the very different setting of a Behavioural Therapy Unit: This could be amplified in a unit such as ours that is staffed by several part-time consultants, arguably making the logistics of culture change even more difficult.

‘It takes a clinical unit at least a year to implement outcome measurement to the point where clinicians do it as a routine…Implementation of any audit imposes a way of working and values implicit in the measures chosen and rated’ (Marks, 1998: 283).

In summary, the aim of the study was met in that a pilot cycle of clinical audit was set up to monitor outcome in our psychotherapy department. Understandably, numbers were small and the numerical results of little statistical value. What is more interesting was the process itself and the problems that we uncovered in relation to setting up a more substantial project. With careful consideration of such difficulties, with perseverance and with forward planning, it is hoped that a complete audit cycle can be established. At the time of writing, over a year after we began, a version of our original audit has been implemented as routine in the department.