Workplace-based assessments in Wessex: the first 6 months

Abstract

Aims and Method

We surveyed educational supervisors and trainees in Wessex about their experience of the first 6 months of using workplace-based assessments (WPBAs), to see whether they needed further support in using them and, if so, in which areas. An anonymous questionnaire was sent to all trainees and educational supervisors in Wessex.

Results

Overall, 63% of trainees and 61% of educational supervisors responded; 22% of supervisors had not received training in WPBA and 61% of trainees identified barriers to completing it. Non-medical staff were rarely approached for assessments.

Clinical Implications

There is a need for further training of supervisors, a more user-friendly IT system and expansion of the role of non-medical staff as assessors.

Over recent years, there have been significant developments in the education, assessment and appraisal of doctors in training in the UK. The establishment of the Postgraduate Medical Education and Training Board (PMETB), Modernising Medical Careers (MMC), and other drivers for change have seen the growth of a new unified training grade in medical specialties and a new competency-based curriculum in psychiatry. ¹ The Chief Medical Officer for England's report, Unfinished Business, highlighted the lack of regular appraisal and formal assessment of trainees’ ‘performance’ and pointed out that the trainee's progression through training grades was largely dependent on examinations. ² He proposed ‘competency-based assessment throughout training’ to be quality assured by a new PMETB.

The Royal College of Psychiatrists’ competency-based curriculum for psychiatry specialty registrars was approved by PMETB at the end of 2006. Following a pilot programme across several sites, this curriculum and workplace-based assessments (WPBAs) were implemented throughout the UK in August 2007.

As of 2007, there were eight WPBAs used in psychiatry: Assessment of Clinical Expertise (ACE), mini-Assessed Clinical Encounter (mini-ACE), Case-Based Discussion (CBD), Direct Observation of Procedural Skills (DOPS), Case Presentation (CP), Journal Club Presentation (JCP), Assessment of Teaching (AoT) and mini-Peer Assessment Tool (mini-PAT). As a minimum, the College recommends two ACEs, four mini-ACEs and two CBDs per year. ³

We undertook a survey of educational supervisors and trainees in Wessex to find out their experience of the first 6 months of using WPBAs. The aim was to see whether they needed further support in using those assessments and, if so, in which areas.

More specifically, we looked at who was carrying out the assessments, which assessments had been used, which were felt to be most useful in developing clinical skills and whether there were any barriers to completing them. We also asked what training supervisors had undertaken and how confident they were in using the assessments.

Method

We developed anonymous questionnaires that were sent to all educational supervisors and psychiatric trainees (ST1-ST4) in Wessex (Hampshire, Dorset, Wiltshire and the Isle of Wight). Those sent to educational supervisors focused on the training they had received in WPBA tools and their level of confidence in using them, and differed from those sent to trainees where the focus was on the experience of using these tools and their perceived usefulness (for the questionnaires, see the online supplement to this paper).

The contact details of all trainees and educational supervisors were obtained from the postgraduate department at the Royal South Hants Hospital and through the educational programme coordinator at St James’ Hospital, Portsmouth.

All trainees had commenced their posts under the MMC in August 2007 and the anonymous survey was piloted in Dorset in February 2008, before being extended a month later to the other areas. The questionnaires were sent by email to all educational supervisors in Dorset and by internal post in the other areas. Those for the ST1-ST3 trainees were hand delivered at the local Member of the Royal College of Psychiatrists course and for the ST4 trainees they were disseminated by internal mail. In total, the questionnaire was sent to 62 trainees and 65 educational supervisors. For questionnaires sent by internal post, a self-addressed envelope was included. In all cases, reminders were sent after about 3 weeks.

The questionnaire comprised questions requiring a ‘yes’ or ‘no’ answer, questions on a 5-point Likert scale and those inviting free text responses. Data were analysed using descriptive statistics by our trust's audit department.

Results

Out of 62 trainees approached, 39 returned the completed questionnaires (63% response rate) and out of 65 educational supervisors, 40 completed and returned the questionnaires (61% response rate).

Responses from educational supervisors

Among the educational supervisors, 31 (78%) had received some form of training in WPBAs. Of those, 6 had received training from the Royal College of Psychiatrists (20%) and 24 from local educational leads (80%). Most (83%) of those who attended training found it to be useful or very useful, 7% did not find it very useful and 10% were ambivalent.

Of the supervisors who had been trained, 55% felt confident or very confident in undertaking WPBA with trainees, 35% felt ambivalent and 9% felt unconfident or very unconfident. Among those who had not received training, only 33% felt confident or very confident in using these tools.

We were interested to know how many WPBAs had actually been performed in the first 6 months since the introduction of the new MMC curriculum. Unsurprisingly, educational supervisors seem to have done the most assessments with their own trainees. In the first 6 months, 63%, 60% and 55% had done between 1 and 3 each of CBD, mini-ACE and ACE respectively. In general, mini-PAT, JCP and AoT seem to be those carried out least often with one's own trainee (percentage of supervisors who carried out none of these with their own trainees was 68%, 50% and 75% respectively).

Few educational supervisors had carried out assessments with other trainees (about 20%): 80% had done no ACEs with other trainees, 70% had done no CBDs, and 65% had done no mini-ACEs. Our survey found that educational supervisors are more likely to be asked to complete a mini-ACE (28%), JCP or CP (25%), or a mini-PAT (25%) with another trainee than any other WPBA. Table 1 details the number of WPBAs undertaken by educational supervisors with their own trainees and with others.

Table 1. Details of the numbers of workplace-based assessments undertaken by trainees and educational supervisors

	Trainees (n=39)				Educational supervisors (n=40)
	With own supervisor, n		With other members of staff, n		With own trainee, n		With other trainee, n
Type of WPBA	0a	1–3b	0a	1–3b	0a	1–3b	0a	1–3b
CBD	6	30	11	7	8	25	28	7
mini-ACE	18	21	9	9	8	24	26	11
ACE	17	23	10	8	17	22	32	8
TAB/mini-PAT	29	10	12	3	27	11	29	10
JCP or Case Presentation	20	18	9	9	20	15	26	6

Our survey found that educational supervisors who had not received training in the use of WPBAs were just as likely to complete them as those who had received training (89% v. 93%).

A significant minority felt they needed more training in most of the WPBAs (Box 1): 40% needed training in CBD, another 40% in ACE, 38% wanted training in mini-ACE, and a further 38% in Team Assessment of Behaviour (TAB) or mini-PAT, 40% wanted training in AoT and 35% in JCP or Case Presentation Assessments.

Some comments from educational supervisors included the following.

1. • ‘I do not think the training is at fault, I have more difficulty just getting used to putting things into practice’.

2. • ‘My own view is that I doubt that this is a fair way to assess junior doctors but I will reserve judgement until [it has been] up and running for longer and I hope to get grips [with it]’.

3. • ‘The main benefits of WPBA are direct observation of trainee and formative use, i.e. clear feedback’.

4. • ‘The WPBA were another drain on time, and combined with the difficulty arranging them in out-patient clinics, contributed to me refusing to have a trainee anymore’.

5. • ‘I think WPBA are an excellent training tool provided they are not used in isolation from other assessment methods. They help you to assist weak trainees, identify areas of difficulty which can be re-assessed with further WPBAs’.

6. • ‘[It is] taking much longer to feel I am doing everything’.

Box 1. Forms of training suggested by educational supervisors and the number that made each suggestion

1. • Workshop/seminar (5)

2. • Practical hands-on marking (5)

3. • Experience/discussion (3)

4. • Video/DVD (2)

5. • In written format (2)

6. • More obvious standards/guidelines (1)

7. • 1 hour at lunchtime (1)

8. • Lecture (1)

9. • Question and answer session(1)

10. • Short update session on teething problems with the process (1)

11. • Manual setting out expectations (1)

12. • Local consultant specific training (1)

Responses from trainees

The completed questionnaires from the trainees showed that most WPBAs were carried out with their educational supervisors, but 49% had carried out WPBA with other members of medical staff (other consultants, specialist registrars, ST4 and associate specialists). We found that no WPBAs other than mini-PAT were done with non-medical members of staff.

Details of the number of WPBAs undertaken by trainees with their own educational supervisors and other members of staff are presented in Table 1.

When asked whether they perceived these tools of assessment as being useful in the development of their skills as a clinician, only 39% of trainees found them useful or very useful, whereas 32% found them useless or not very useful and 29% were ambivalent. It would be important to find out why trainees felt this way. The most useful of the WPBAs were CBD (38%), followed by the ACE (26%). On the CBD, trainees commented that it was ‘most relevant to practice’, ‘allocates time for discussion and ensuring understanding’, and ‘helps understand about management of complex patients’.

The ACE, on the other hand, was valued for its more longitudinal view, observation with the patient and as good practice preparation for the old part 2 exam which was still being used at that time. Comments from trainees about the ACE included the following.

1. • Allows more longitudinal view.

2. • Observed with patient.

3. • Checks clinical history is taken.

4. • Able to get fair feedback and identify shortcomings.

5. • Identifies strengths and weaknesses.

6. • Good practice tool to prepare for part 2.

Other comments from trainees about the WPBAs included that ‘all are useful as they give ideas about your strengths and weaknesses’ and ‘[JCP are useful because] of research to be done to get a good paper’.

Barriers to completing WPBAs

We asked trainees whether they had faced any barriers to completing the assessments and most (61%) did identify barriers, roughly split into problems caused by lack of training (both trainee and trainer), problems with IT and time constraints (Box 2).

The survey also looked at the percentage of trainees and supervisors who had registered with Healthcare Assessment and Training (HcAT) at that stage. Only half of educational supervisors had registered with HcAT. Those who had registered commented that it was cumbersome and confusing to get around and that they had problems with the website. We understand that the website (https://training.rcpsych.ac.uk/assessments-online) has improved since.

A greater number of trainees (76%) had registered with HcAT, but only about half of those (49%) had completed online assessments. Some comments on barriers to completing those assessments were: ‘I could not get it to work’, ‘Complicated with no clear instructions’, ‘Local IT network does not support it’.

Box 2. Barriers to use of workplace-based assessments according to trainees

Barriers due to lack of training

1. • Uncertain which to perform and how many at ST4 level

2. • Lack of information about them

3. • Consultant had limited knowledge of assessment tools

4. • Consultants opposed to process

5. • Confusion over requirements

6. • Other non-medical members not trained and do not feel confident to do it

7. • Using e-portfolio

Barriers due to IT problems

1. • IT problems - unable to enter assessment online

2. • Website is badly designed with poor instructions

3. • Difficult on line logging in, assessors not completing electronic assessments

4. • No follow-up of results

Barriers due to time constraints

1. • Lack of time (trainee or consultant)

2. • Finding an assessor

3. • Workload

Discussion

Our survey concerned the practical implementation of the WPBAs in Wessex and the initial impressions of trainees and educational supervisors about these tools of assessment; it was not about the validity or reliability of these tools in the assessment of competence. The response rate from trainees and educational supervisors was 63% and 61% respectively. This may well have resulted in bias as in fact the non-responders may not have completed any WPBAs or were even less confident about the use of these tools.

Trainees’ views on WPBA usefulness

We found that only 39% of trainees felt that WPBAs were useful or very useful in developing their skills as a clinician. This clearly brings to mind further questions as to whether these assessments are seen by trainees as formative rather than summative tools or whether trainers are maximising the potential of WPBAs in giving useful feedback.

Trainees only approached medical staff rather than other professionals for all the assessments apart from mini-PAT. This could be caused by a lack of confidence among non-medical staff about undertaking the assessments or trainees not considering these members of staff as being able to give useful feedback. This may imply the need for more effective training and identification of senior non-medical staff who are able to carry out those assessments.

One of the concerns raised by this survey was that trainees did not appear to be undertaking the minimum number of WPBAs as recommended by the College for each year of training. Since conducting the survey, the Wessex trainees have had their Annual Reviews of Competency Progression (ARCP) and only six of them (ST1-ST4) had submitted the College recommended number of WPBA at the time of the paper review of the ARCP. This may well be a reflection of the ongoing need for training of assessors, identification of non-medical assessors and advance planning by trainee and supervisor so that there is adequate time to complete the required number of assessments. Alternatively, it may reflect the current hybrid use of paper and electronic storage of the WPBA with poor access to the latter during the ARCP.

Some of the barriers identified by trainees in undertaking the assessments clearly highlight the need for more training of the educational supervisors and this undoubtedly will have impact on the use of these assessment tools as being a more worthwhile experience for trainees. The online assessments need to be user friendly and there should be an improved IT system if more WPBAs are required to be completed online.

The trainees found CBD and ACE to be the most ‘useful’ of the WPBAs, even though more mini-ACEs were completed. This could be due to the trainees gaining more in-depth knowledge about the assessment and management of individual cases as, in general, more time is allocated for these assessments.

There are questions as to whether the evidence from these assessments can indeed inform summative decisions regarding the trainee's progress. We believe that in a curriculum for postgraduate medical education where the expectation is for trainees to be active learners (educational supervisors being facilitators), assessments, when used effectively, can provide a true reflection of the trainees’ achievements, thus being the foundation for further educational activities. Assessment could then be viewed as a formative process, with a number of such assessments offering a pattern of evidence across a period of learning. The analysis of this evidence could then provide a robust and fair ‘summative statement’ about what the trainee has achieved and how they have developed. ⁴

Conclusion

The findings of our survey are purely representative of the experiences of the trainees and educational supervisors in Wessex during the first 6 months of WPBA implementation. It would be interesting and informative to have further surveys conducted in other areas to see whether the findings are unique to Wessex or are in fact echoed elsewhere.

Our survey found that non-medical staff were seldom involved in assessments apart from mini-PAT. There is little evidence on the effect of using assessors who are not senior medical clinicians to assess experienced postgraduate trainees for a wide variety of competencies. ⁵ However, with the College expecting non-medical staff to be assessors (https://training.rcpsych.ac.uk/assessments-online-guidance), it would be important to investigate barriers to this and the means to overcome them.

Since the survey, there has been further training in the Wessex region in the use of WPBAs, well-attended by medical and non-medical staff. The Wessex School of Psychiatry has also taken steps to ensure that trainees have been made aware of the College requirements through adequate correspondence to relevant tutors.

We are concerned that 22% of educational supervisors had not received any training in WPBAs but were still carrying them out 6 months after their implementation. We feel that one way of ensuring uniformity in training of supervisors, therefore improving the quality of assessments, would be to incorporate on the College website training videos that could be accessed locally by those involved in education and training.

It is vital to try and develop enthusiasm among trainees and trainers in developing adequate experience in using these tools of assessment, supported by a well-functioning IT system. If this could happen there is a real chance that the use of these tools could be truly educational and worthwhile experiences.

Declaration of interest

None.