Case-based discussion: a useful tool for revalidation

Abstract

Aims and method

Revalidation is the process by which doctors demonstrate that they are up to date and fit to practise. Case-based discussion has been identified by both psychiatrists and service users and carers as a potentially useful tool for revalidation. The aim of this study was to examine the feasibility of case-based discussion for assessments to use in revalidation. A pilot using case-based discussion as an assessment/developmental technique for revalidation was undertaken in six trusts involving 86 consultant psychiatrists.

Results

The average time taken for each case-based discussion including preparation and reflection was 48 min. All but one of the doctors involved in the case-based discussion pilot described the process as useful or very useful in improving clinical practice. In total 87% of the assessed doctors and all of the assessors reported that they considered case-based discussion to be a useful part of the revalidation process assessed. The majority of doctors assessed believed that cases should be chosen at random (91%).

Clinical implications

Case-based discussions have the potential to be a useful quality improvement tool for revalidation.

Revalidation is the process by which doctors will have to demonstrate to the General Medical Council (GMC) that they are up to date and fit to practise and that they are complying with the relevant professional standards. Since autumn 2009 doctors wanting to practise medicine in the UK not only have to be registered with the GMC but also have to hold a licence to practise. All the professional activities that were restricted by law to doctors who are registered with the GMC will be restricted to doctors who hold a licence. These activities include prescribing, signing death and cremation certificates and holding certain medical posts in the National Health Service (NHS) and the independent sector.

All doctors must practice in accordance with the generic standards of practice based on the GMC's guidance Good Medical Practice. ¹ Practising doctors who undertake specialist practice continue to meet the particular standards that apply to their medical specialty or area of practice. These specialty-specific standards are set by the medical Royal Colleges and faculties. For psychiatrists, the relevant standards are set out in Good Psychiatric Practice (Third Edition). ² All doctors will be required to collect a portfolio of supporting information that will be reviewed at appraisal on an annual basis. Revalidation is planned to take place on a 5-yearly cycle and hence the necessary evidence to provide assurance about keeping up to date and fit to practise can be gathered over a 5-year period.

The key process through which revalidation standards will be demonstrated is appraisal. The NHS is piloting an enhanced appraisal process that will have both summative and formative components. ³ The summative component of appraisal will involve looking back at what has been achieved and the formative part will be the agreeing of a personal development plan outlining the way forward.

The Royal College of Psychiatrists considered a variety of methods by which the clinical practice of those psychiatrists no longer in training and on the specialist register could be assessed within the context of revalidation. An online consultation process was undertaken in which College members were asked to rate their support for a range of different measures. These measures were:

1. • audits of practice against guidelines

2. • outcome measures

3. • written tests of knowledge

4. • online teaching modules with assessments

5. • case-based discussion

6. • direct observations of practice - observing ward rounds

7. • direct observation of practice - observing out-patient clinics

8. • multisource feedback from colleagues and patients.

A total of 106 psychiatrists responded to the online consultation and the results for each of the methods is shown inTable 1.

Table 1 Online consultation results

	%
Question	Strongly disagree	Disagree	Neither agree or disagree	Agree	Strongly agree
I support the use of standardised audit measures to increase the reliability and validity of appraisal	4	15	34	42	5
I support the use of clinical outcome measures to increase the reliability and validity of appraisal	4	23	32	37	4
I support the use of written knowledge tests as a way of demonstrating adherence to College standards	32	36	22	6	3
I support the use of online assessment modules with questions as tests of knowledge	17	14	25	42	3
I support the use of a form of case-based discussion as a way of demonstrating adherence to College standards	8	9	22	57	4
I support the use of direct observation of practice as a way of demonstrating adherence to College standards	16	18	27	32	6
I support the use of multisource feedback from patients as a way of demonstrating adherence to College standards	9	12	14	42	22
I support the use of multisource feedback from multidisciplinary colleagues as a way of demonstrating adherence to College standards	4	5	17	45	29

The College also asked two focus groups of users and carers (24 individuals in total) to consider the possible methods of assessing specialist practice and the possible processes for use in revalidation. The ranking is shown inBox 1 (where 1 is most favoured and 8 is least favoured).

Box 1 Ranking by users and carers (n =24)

Case-based discussion

1

Directly observing practice: ward rounds

2

Anonymous feedback from service users, carers and colleagues

3

Directly observing practice: out-patient clinics

4

Online modular assessments

5

Outcome measures

6

Audit of practice against guidelines

7

Written tests

8

It can be seen from these results that the most favoured method, both by psychiatrists themselves (other than multisource feedback) and by the users and carers, for use in revalidation was case-based discussion. Multisource feedback is likely to be recommended by the GMC as an essential component of revalidation. ³ It was decided to undertake a pilot to assess the potential value of case-based discussion in revalidation.

What is case-based discussion?

Case-based discussion involves a doctor reviewing the assessment and treatment of a patient provided by a colleague. Diagnosis, assessment and management are reviewed. This is done in discussion between the doctors based on what is recorded in the case notes. However, what is written down can be explained and amplified by the treating doctor, allowing a much fuller picture to be obtained than could occur from a simple case-note review. The process can be used to discuss a doctor's reasoning and judgement. Although the process is largely formative, it can have an assessment component.

The system piloted for psychiatrists on the specialist register has been adapted from that used by trainees ⁴ to meet the different requirements of more experienced doctors.

Method

The case-based discussion pilot study was based in six English trusts from January 2008 to April 2009. Participating trusts were: Berkshire Healthcare NHS Foundation Trust, Cornwall Partnership NHS Trust, Dorset Healthcare NHS Foundation Trust, Mersey Care NHS Trust, Northumberland Tyne & Wear NHS Trust, 2gether NHS Foundation Trust for Gloucester. All consultants in each trust were invited to participate in a training workshop run in each of the organisations explaining both revalidation and case-based discussion. Written guidance on how to undertake a case-based discussion with a consultant colleague was provided. It was emphasised in the training that the main purpose of the case-based discussion was formative, to determine both areas of good practice and areas for improvement. The areas for improvement would then link into a personal development plan.

Standards assessed for case-based discussion were taken from Good Psychiatric Practice. ² These standards were in eight domains:

1. 1 assessment

2. 2 diagnosis

3. 3 risk assessment

4. 4 treatment plan and delivery

5. 5 knowledge of treatment options

6. 6 record keeping

7. 7 communication with professionals

8. 8 communication with patients.

A full description for each standard taken from Good Psychiatric Practice was provided on the assessment form. Possible ratings were as follows:

1. • insufficient evidence/not applicable

2. • 0: fails to meet standards of Good Psychiatric Practice

3. • 1: significant inconsistency in meeting standards of Good Psychiatric Practice

4. • 2: some inconsistency in meeting standards of Good Psychiatric Practice

5. • 3: meets standards of Good Psychiatric Practice and consistent with independent practice

6. • 4: exceeds standards of Good Psychiatric Practice

7. • 5: excels standards of Good Psychiatric Practice.

Advice as to practical arrangements was as follows.

1. 1 Arrange for someone to select, at random, two sets of case notes of patients whom you saw during the previous 2 weeks.

2. 2 Choose one of the two patients for discussion - a case that you believe will best assist improving your practice.

3. 3 The assessor should have the opportunity to review the case notes prior to the assessment.

4. 4 Both participants should consider beforehand what are the key points for discussion.

5. 5 It was recommended that each discussion would take 20-30 min and should occur with the case notes to hand.

6. 6 The discussion should commence from the doctor's entry (or entries) in the case notes. The following can be used as a guide to frame discussion:

   1. • general introduction

   2. • assessment/diagnosis

   3. • investigations/referrals

   4. • therapeutics

   5. • follow-up/care plan

   6. • monitoring chronic illness

   7. • individual patient factors

   8. • care setting.

7. 7 Feedback should follow immediately after the case-based discussion and it was recommended that self-assessment should be encouraged, focusing on both positive aspects of performance and areas of concern.

8. 8 An action plan for any continuing professional development needs should be formulated.

Once the case-based discussion had been completed, participants were asked some general questions on the case-based discussion process. The first question was: How useful was the discussion in helping improve your clinical practice? Rated:

1. • 0: not at all useful

2. • 1: of little use

3. • 2: of some use

4. • 3: useful

5. • 4: very useful.

The second question was: I consider case-based discussion is a useful technique as part of the revalidation process. Rated:

1. • 0: strongly disagree

2. • 1: disagree

3. • 2: neither agree or disagree

4. • 3: agree

5. • 4: strongly agree.

Results

One hundred and ten consultants attended the six workshops. Participants arranged sessions with colleagues over a 4-month period between January and May 2009 and undertook between one and four case-based discussions. It was suggested that ‘assessors’ swapped places with ‘assessees’ in order to maximise the effectiveness of the sessions. In addition to the case-based discussion assessment form, feedback about the process was completed by each participant. Forms were faxed back to the College and results collated. This paper presents an analysis of scores and participant feedback from the case-based discussions held.

In total there were 140 case-based discussions scored, recorded and analysed, involving 86 assessed doctors and 86 assessors. Discussions were scored on a 0-5 scale over eight domains, with higher scores reflecting better performance. Table 2 shows the average score on each of the domains. Table 2 also shows the ‘corrected item-total correlations’ that gives an idea of how well each domain correlates with the total score. Item-total correlations range from 0 to 1 with smaller values reflecting less discriminating domains. Out of all the domains scored, the ‘risk assessment’ score correlated least with the total score but not worryingly so. Scoring was highly consistent over the eight domains (Cronbach's alpha = 0.94) suggesting that scoring was reliable within a discussion and reflects one underpinning domain. It is noteworthy that five out of the eight domains were never marked less than three by an assessor.

Table 2 Feedback scores given over the eight domains

	Mean (s.d.)	Range, 0-5	Corrected item-total correlation	α
1 Assessment	3.69 (0.75)	3-5	0.86
2 Diagnosis	3.62 (0.70)	3-5	0.80
3 Risk assessment	3.38 (0.71)	2-5	0.65
4 Treatment plan and delivery	3.66 (0.76)	2-5	0.84
5 Knowledge of treatment options	3.69 (0.72)	3-5	0.82
6 Record keeping	3.48 (0.74)	2-5	0.72
7 Communication with professionals	3.71 (0.75)	3-5	0.84
8 Communication with patients	3.69 (0.71)	3-5	0.79
Overall	3.60 (0.60)			0.94

Doctors assessed on more than one case-based discussion

Twenty-one doctors were assessed on more than one case. The average score over the eight domains significantly increased with the number of attempts (F(3,24) = 15.3, P<0.001) suggesting an improvement with practice.

Areas identified as good practice and suggestions for development

In total, 137 of the 140 case-based discussions had free-text comments identifying examples of good practice. Analysing the free-text comments demonstrated that the areas most commonly included as good practice points were as follows:

1. • competent assessment, 38%

2. • communication with patients, 26%

3. • communication with professionals, 24%

4. • treatment plan and delivery, 23%

Free-text comments on areas for development were present in 105 of the 140 sets of case notes. Analysis of these comments show that the most common areas of suggestions for development were as follows:

1. • competence of assessment, 13%

2. • communication with patients, 13%

3. • communication with professionals, 9%

4. • record keeping, 8%

5. • treatment plan and delivery, 8%

In total, 17% of the case-based discussions completed had a single comment stating that practice was either strong in all areas or that no areas of weakness were identified.

Although most comments were specific to the given case there are some similarities that can be drawn out. For instance, many of the comments under ‘areas identified as good practice’ referred to broad, thorough or longitudinal assessments (51 comments), patient concern and/or communication (20 comments), evidence of multidisciplinary working (16 comments) and good record keeping (16 comments).

Performance of assessors

Twenty assessors scored more than one case-based discussion. Some assessors had very low variance in their scores. Less variation may reflect an assessor who marks within a restricted range or may be because the standard of the doctors did not show much variation. Establishing the reliability of assessors is difficult as both the case and the doctor vary between assessors. With more data it may be possible over time to assess the consistency of marking between assessors.

Analysis of feedback

After the case-based discussions, participants were asked to fill out a feedback form on which they were asked to state how long each case had taken them to complete. Table 3 shows a breakdown of this data. On average, case-based discussions took 48 min to complete (11 min in preparation, 30 min for the discussion and 8 min in reflection). There were no significant differences between assessed doctors and assessors in their reports of how long the components of a case-based discussion took.

Table 3 Feedback on time spent on each part of case-based discussion

	Mean (s.d.)
Time, minutes	Assessed doctor (n = 46)	Assessor (n = 36)	All (n = 88)
Preparation	11.7 (14.7)	9.8 (10.3)	10.9 (12.6)
Duration	27.8 (12.6)	31.4 (13.7)	29.9 (13.5)
Reflection	8.5 (4.8)	7.6 (6.0)	8.2 (5.4)
Whole process	47.3 (23.7)	47.8 (20.9)	48.2 (22.7)

Feedback data are displayed in Table 4. All but one of the assessed doctors said that they found the process useful or very useful in improving clinical practice, as did all the assessors. The majority of assessed doctors (86.8%), and all of the assessors, reported that they considered the case-based discussion to be a useful part of the revalidation process.

Table 4 Feedback on usefulness of case-based discussion

	Mean (s.d.)
	Assessed doctor	Assessor	All (n = 88)
Improving clinical practice	3.2 (0.79)	3.0 (0.67)	3.1 (0.74)
Part of the revalidation process	3.1 (0.78)	3.2 (0.61)	3.1 (0.69)

When asked how many case-based discussions they thought should be completed over a 5-year period the most common response from the assessed doctors was 5 (47%, mean 6.88, s.d. = 4.60), whereas the most common response from assessors was 10 (43%, mean 7.57, s.d. = 3.52). The majority of participants who expressed a view indicated that the cases should be chosen at random (91%).

Discussion

The formal and informal feedback on the usefulness of case-based discussion indicated that 85 out of 86 doctors involved thought that the process was useful or very useful in improving clinical practice, and the vast majority (87%) thought that case-based discussion could be a useful component for revalidation. This result supports the findings from the initial surveys as to the potential usefulness of case-based discussion and provides a sound basis for the College recommending that case-based discussion be included as a component for revalidation.

The positive feedback for the assessment tool expressed by the doctors in this study is in marked contrast to the resentment and mistrust expressed by UK trainee psychiatrists towards workplace-based assessment. ^4-6 This may be because the participants in this study were volunteers and attitudes to case-based discussion may be less benevolent if it became a compulsory component of assessment and it is seen more as a summative tool.

It is important to emphasise that the main use of case-based discussion is as a developmental process. It is this process that specialists valued and found useful. The assessment component of case-based discussion, although having a potential role, in this pilot rarely identified practice that was less than satisfactory. This may reflect that no practice was less than satisfactory but equally may reflect a reluctance on the part of colleagues to rate any aspect of their colleagues’ practice as unsatisfactory. The ability and willingness to do this should be an important part of training for case-based discussion. The rating of an area as unsatisfactory does not mean that the doctor's overall performance is unsatisfactory. Even in that particular area, the case chosen may be an isolated example. Further work will need to be done to give examples to colleagues as to when a rating of unsatisfactory might be used. The fact that there were no ratings of 0 and 1 on the scale suggests that these two boxes could be combined as there is no evidence that the graded rates of unsatisfactory performance are likely to be used.

Further work needs to be done on training assessors in how to assess in order to ensure the confidence of the profession in the process. The emphasis must remain on the use of the process to identify developmental needs. Case-based discussion is likely to be of greatest benefit where both assessors and those being assessed are trained, not only in the standards required, but also in how to provide and accept developmental feedback from a colleague.

The issue of how cases are selected is important. Most doctors in the study supported random selection. However, if case-based discussion is to be used by doctors to demonstrate fitness to practise, there may need to be an opportunity to select occasional cases in order for the process to review all aspects of the doctor's case-load. Random sampling may not identify an appropriate spectrum of cases.

It is recognised that case-based discussion is not the only workplace-based assessment that might be used to support revalidation. At this stage, however, it will be the only one to be recommended by the Royal College of Psychiatrists. It is a tool that will enable specialists to demonstrate that they meet several of the standards of Good Psychiatric Practice ² and also has an important developmental role.