Survey of non-medical professionals on the use of workplace-based assessments

Abstract

Aims and method

Non-medical staff are eligible to assess trainee doctors through mandatory workplace-based assessments (WPBAs). An anonymous questionnaire was given out to non-medical staff working with trainees in community and in-patient settings at Royal Blackburn Hospital. Our aims were to look at their awareness of and familiarity with assessor guidance, trainee competencies, training needs and assessors' views on completing these assessments.

Results

In total 118 of 150 (79%) individuals returned a questionnaire and 89 WPBAs had been carried out. Most assessors were Band 6 (or equivalent) or below (53%). Most assessors had neither read any assessor guidelines (75%) nor were familiar with the competencies required of a doctor (76%). Although 79% felt that non-medical staff should be assessing trainee doctors, only 44% felt comfortable doing this. None had been trained and 92% felt this would help. Twenty WPBAs (excluding mini-peer assisted tools) were carried out by staff at Band 6 or below.

Clinical implications

No respondents received guidance or training on being an assessor. This highlights the need for urgent action and delivery of training. This can easily be adapted from training packages developed for medical staff.

Workplace-based assessments (WPBAs) have been introduced throughout the postgraduate medical disciplines, as part of the overhaul in assessment of doctors by the Postgraduate Medical Education Training Board (now part of the General Medical Council). Many trainees spend much of their time on the ‘shop floor’ where professionals other than medics observe their work. Psychiatrists utilise several WPBAs, with at least nine in use by the North Western Deanery and Royal College of Psychiatrists in 2010 (see Appendix). The Royal College of Psychiatrists recognises that assessments from ‘a range of different assessors in different situations’ lead to a more accurate picture of the trainee's competencies as it ensures triangulation of evidence. ¹ Previous research has commented on the importance of trainee education in appreciating feedback from non-clinical sources. ² The College's guidance ³ is quite clear that the only non-medical staff who should be assessing specialist trainees in years 1-3 (ST1-3) and core trainees in years 1-3(CT1-3) are ‘Band 7 nurses or psychologists or the equivalent’. Similarly, ST4-6 doctors can only be assessed by ‘Consultants and Band 8 nurses or psychologists or the equivalent’. The above information does not apply to mini-peer assisted tools (mini-PATs). Surveys have been carried out on the attitudes of consultants, specialty doctors and trainees to WPBAs, ⁴ but to our knowledge, very little has been written about the assessments of doctors by non-medical staff (such as nurses, social workers, occupational therapists, psychologists, etc.), and their views about the process. Other studies have called for ‘further robust investigation to examine questions of confidence in the workplace based assessment’. ⁴

Our aims were to look at the awareness levels of guidance available to assessors and familiarity with trainee competencies, adherence to Royal College guidance and whether there is any need for further training for this group of potential assessors.

Method

An anonymous paper-based questionnaire survey was given out to non-medical staff working with trainees in psychiatry at Royal Blackburn Hospital (online supplement). Community and in-patient units were covered, both in old age and general adult psychiatry. Non-medical staff included nurses, occupational therapists, psychologists, social workers, administrative staff, community psychiatric nurses and support workers. Students were also included as they could potentially carry out mini-PAT assessments. The trainees who were being assessed included foundation doctors, general practitioner (GP) trainees, ST1-6, and CT1-3 psychiatry trainees.

Views of the staff were ascertained by way of free text boxes. One of the authors (N.H.) read all the text and used thematic analysis to group any recurrent themes together. The Trust Research and Development department confirmed in writing that ethical approval was not required for this study.

Results

A total of 118 staff returned the questionnaire, out of 150 that were given out, representing a 79% return rate. The pie chart (Fig. 1) demonstrates a break down of assessors by banding seniority (or their equivalent). Of the responders, 25% (n = 31) rated their position as ‘other’ and of these, 17 were support workers, 8 were administrative staff (medical secretaries), 4 identified themselves as ‘Band 3’ staff, 1 was an assistant psychologist (Band 5), and 1 was a nursing student. The only WPBAs this group of staff completed were mini-PATs.

Fig 1 Seniority of non-medical staff completing workplace-based assessments.

Figure 2 depicts the responses on the questionnaire. A confirmed total of 89 WPBAs were completed (Fig. 3). Out of these, mini-PATs were the most common (n = 21, 24%), followed by mini-assessed clinical encounters (mini-ACEs) (n = 13, 15%). There were 7 case presentations, 5 ACEs, 3 directly observed procedural skills, 3 direct observation of non-clinical skills, 3 assessments of teaching and 1 case-based discussion.

Fig 2 Responses to the survey by non-medical staff assessors (n = 118).

Fig 3 Number and type of workplace-based assessments (WPBAs) where the assessor was a non-medical member of staff. Mini-PAT, mini-peer assisted tool; mini-ACE, mini-assessment of clinical expertise (looking at one aspect, e.g. risk assessment or mental state); CP, case presentation; ACE, assessment of clinical expertise (looking at whole assessment, e.g. full history and examination); DOPS, direct observation of procedural skills (e.g. electroconvulsive therapy); DONCS, direct observation of non-clinical skills (e.g. chairing meetings); AoT, assessment of teaching; CbD, case-based discussion.

A total of 65 trainees were assessed, which included GP trainees (n = 15), foundation year trainees (n = 11), CT1-3/ST1-3 (n = 23) and ST4-6 (n = 16).

Workplace-based assessments, other than mini-PATs, were filled out by staff who identified themselves as ‘Band 6 or below, or its equivalent’. These included Mini-ACEs (n = 10), case presentations (n = 3), directly observed procedural skills (n = 2), case-based discussion (n = 1), assessments of teaching (n = 1) and direct observation of non-clinical skills (n = 1). The trainees being assessed by this group included ST1-3/CT1-3(n = 12), ST4-6(n = 7), GP trainees (n = 6), and foundation year trainees (n = 2). Similarly, the trainees assessed by staff who were ‘Band 7 or equivalent’ included ST1-3/CT1-3 (n = 4), GP trainees (n = 3) and foundation year trainees (n = 4). No ST4-6 were assessed by this group.

Responses in the free-text section of the questionnaire can be found in Box 1. After analysis of all the free text, the themes that emerged were as follows.

1. • Non-medical staff do have a role in assessing trainee doctors.

2. • Assessor training is needed.

3. • Many assessors feel uncomfortable assessing aspects that they do not carry out themselves.

Discussion

We believe that this is one of the first studies looking at the views of non-medical staff about the mandatory WPBAs for trainees. This is interesting as, arguably, trainees are spending more time working with non-medical staff since the introduction of the European working time directives and new ways of working such as ‘hospital at night’. ⁵ Apart from the case-based discussions and mini-PATs, all other WPBAs require the presence of an assessor at the time of assessment. It is therefore envisaged that more trainees will be requesting non-medical staff to complete their WPBAs.

Our study had a respectable response rate of 79% compared with other studies on WPBAs, ⁴ although it was confined to one hospital covering two psychiatry subspecialties. This survey showed that junior staff (defined here as Band 6 or below, or equivalent) were carrying out assessments, demonstrating a non-adherence to College guidance. ³ Indeed, most of the mini-ACEs carried out on all trainees were assessed by junior staff (10 out of 13). There are reports that staff that are more junior are more likely to give more favourable responses to trainees on their assessments, and also their profession can significantly influence the assessment, ^2,6 which could be a reason why trainees are approaching them.

Box 1 Free-text answers

‘Non medical staff are valuable in assessing other areas of practice, e.g. manner, social, interaction, knowledge of M.H.[mental health] problems, etc.’

‘Trainee doctors have a significant impact on our own work, so in my view it is valid and useful for us to comment on their practice.’

‘As qualifiedand experienced practitioners, we have a validpoint of view on doctors'practice.’

‘My assessment is not necessarily the same as assessing them against the formal standards of the Royal College. I am not sure that this formal assessment should be my responsibility, but if it were, I would need training first.’

‘Non-medical staff may be in a good position to assess other competencies (e.g. interpersonal skills, etc.).’

‘It is important to involve other disciplines in the assessment process. However, it is perhaps not appropriate for non-medical staff to assess clinical competencies.’

‘I feel that assessment of junior doctors by nurses wouldbe beneficial as most of them do not have understanding of mental health. I feel this would improve documentation and quality of health and social needs assessment.’

‘It would be interesting for nurses to receive training regarding assessment of junior doctors to develop on-going support and guidance for both the nurse and doctor.’

‘There is a role for non-medical assessors in the assessment of trainee doctors. However, I think this role needs some definition by both the trust and the particular organisations requesting the assessments.’

‘I do not feel comfortable assessing any other professional who performs any test that I do not personally carry out.’

‘If I am to complete these forms, I should be given extra time to take on extra responsibilities.’

This survey was carried out a few years after implementation of WPBAs, so non-medical staff should have been familiar with the concepts and how to use them correctly. However, only a minority of assessors were aware of guidance, the competencies expected by trainees and the level of performance expected by trainees at different levels of seniority. We believe that any misunderstandings by non-medical staff about the competencies expected of trainees when completing the WPBAs would have biased and invalidated the forms.

Although most respondents stated non-medical staff should be assessing doctors, less than half (44%) felt confident in doing so. This was in part due to many not being sure of what level of competency is expected from a ST2, compared with a ST3 for example. Indeed, a survey of psychiatry trainees’ perceptions found that both medical and non-medical assessors ‘lacked the knowledge and skills necessary to assess them using WPBAs’. ⁷ Vasudev et al ⁷ found that non-medical staff were reluctant to assess trainees and judge whether trainees had reached their expected level of competency. They also concluded that training of assessors remains crucial.

The way forward?

Not a single respondent in our study had received any guidance or training about being an assessor for trainee doctors, but most assessors commented that training would be helpful. This highlights the need for urgent action. The College has already produced a DVD for training doctors in the use of WPBAs. We do not see any reason why these could not be used to train non-medical staff too. Based on this study, we feel the training would need to cover the College guidance, the competencies expected at the different training levels and the value of feedback. Halder et al found that most psychiatry trainees stated that feedback included in the WPBAs was more conducive to learning than the actual scores themselves, and therefore any training needs to highlight this (details available from N.H. on request). However, there needs to be more research on whether the training objectively improves assessments by non-medical staff. Non-medical staff are usually very busy and may not be willing to take on the additional responsibility of being assessors. An ACE, for example, can potentially take an hour and a half for the whole process, including feedback. However, it is worth trying to persuade more non-medical staff to get involved with these assessments, given that there are difficulties persuading medical staff to complete WPBAs such as ACEs. ⁷ If non-medical staff do not get involved then problems may arise with trainees not being able to access enough willing assessors.

Studies have shown that the situation regarding completing WPBAs is improving. Trainees stated that, in 2008, nearly a third of assessors found the forms too complicated and were unsure which WBPA to use for a clinical situation. In the following year, this number had nearly halved for both domains. ⁷ Our study highlights that there is still much to be done, especially with non-medical staff, as WPBAs are here to stay for the foreseeable future.

Appendix

Workplace-based assessments used in psychiatry

Assessment of clinical expertise (ACE)

Mini-assessed clinical encounter (mini-ACE)

Case-based discussion (CbD)

Case presentation (CP)

Journal club presentation (JCP)

Directly observed procedural skills (DOPS)

Mini-peer assessment tool (mini-PAT)

Assessment of teaching (AoT)

Direct observation of non-clinical skills (DONCS)