Medical training in the UK has recently been subjected to radical changes, among which workplace-based assessment (WPBA), introduced in August 2007, is perhaps one of the most controversial. The concept of performance assessment stems largely from concerns about patient safety and a perceived requirement to reassure the public that doctors provide safe, effective and high-quality clinical care. Workplace-based assessment is intended to function as a robust mechanism facilitating regular assessment of trainees along with the provision of specific structured feedback and targeted training, while formally demonstrating such regular review and appraisal. Reference Searle, Holsgrove, Brown and Oakley1
It is of prime importance that any new system of assessment should appear fair, balanced and beneficial to the medical profession. Furthermore, the successful implementation of change is dependent on acceptance of the need for change by and cooperation from those affected by such change. Structured feedback from key stakeholders is in turn vital to the continual refinement of any quality assurance framework. Reference Finucane, Barron, Davies, Hadfield-Jones and Kaigas2 Accordingly, exploration of the attitudes of trainees (and trainers, for whom a separate survey is in progress), acknowledgment of the difficulties and practicalities they face, and measures to augment their understanding of the new process will be crucial to the smooth and effective implementation of WPBA. In the absence of these crucial ingredients, imposition of WPBA upon uninformed and untrained participants could promote inconsistency and non-adherennce, both of which would defeat the very ethos of this new framework.
In this study we therefore set out to explore the attitudes of psychiatric trainees, and the difficulties and practicalities they face with a view to providing the structured feedback so essential to improving the validity, reliability, relevance and practical benefits of WPBA.
Our aims were to explore attitudes and perceptions regarding WPBA among psychiatric specialty registrars appointed under the Modernising Medical Careers (MMC) system in Wales, to identify problems currently experienced and to recommend measures to improve WPBA.
Method
After clearance from the National Research Ethics Service, an anonymous, questionnaire-based, cross-sectional survey was designed with the help of a statistician and a clinical research fellow at the University of Cardiff.
The survey was initially piloted for a week to identify and remedy any potential flaws and was subsequently hosted online, over a 2-month period (6 July-6 September 2008), on the Bristol Online Surveys website (www.survey.bris.ac.uk/). It was conducted under the auspices of the Welsh Division of the Royal College of Psychiatrists and the Welsh Deanery; a comprehensive list of trainees was provided by the latter. All psychiatric specialty registrars in Wales were invited by email to participate.
Many questions were designed as consistently positive statements with regard to WPBA and respondents were required to affirm their agreement or otherwise on a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = no opinion, 4 = agree and 5 = strongly agree). Other questions involved the respondent rating any given subject on a 5-point scale (1 = very poor, 2 = poor, 3 = acceptable, 4 = good and 5 = very good) and the remainder required them to select a preferred answer from multiple choices.
Results
Of a total of 89 psychiatric trainees in Wales, 88 were invited to participate in the study (S.M., one of the researchers, was excluded); 81 participated, a response rate of 92%. There were 22 ST1 trainees (27.2%), 24 ST2 (29.6%), 21 ST3 (25.9%), 12 ST4-6 (14.8%) and 2 doctors at other levels of training (2.5%). Sixty-seven trainees (83%) had trained under the previous senior house officer/specialist registrar training system; 64 of these (79% of the overall total) had done so for over 6 months. The remaining 14 trainees (17%) had not undergone any training under the old system. All but one of the trainees had undergone at least one assessment.
Trainees’ opinions on why WPBA was introduced
Trainees’ perceptions of the main reasons and the most important driving forces behind the introduction of WPBA are presented in Table 1.
Table 1. Trainees’ perceptions about the reasons and driving forces behind the introduction of workplace-based assessment
| % | n | |
|---|---|---|
| Main reasons | ||
| To improve training | 43 | 35 |
| Politically driven | 41 | 33 |
| To improve patient care | 9 | 7 |
| Failure of the previous training system | 5 | 4 |
| Do not know | 2 | 2 |
| Most important driving forces | ||
| PMETB | 44 | 36 |
| Government/political | 44 | 36 |
| Royal College of Psychiatrists | 6 | 5 |
| Do not know | 4 | 3 |
| Public demand | 1 | 1 |
| Trainers | 0 | 0 |
The two most prominent perceived reasons are ‘to improve training’ (43%) and ‘politically driven’ (41%). Though disparate, they account for almost 84% of the reasons given. Only 9% of trainees considered that WPBA was introduced ‘to improve patient care’.
The two most commonly perceived driving forces are the Postgraduate Medical Education and Training Board (PMETB) and the government, identified by almost 89% of trainees.
Attitudes to and perceptions of WPBA
The majority of trainees are unimpressed with WPBA as an educational/assessment tool in terms of its reliability, validity and the evidence that underpins it (Table 2). More than three in four trainees expressed concerns about the perceived impact of assessors’ personal preferences, the provision of training and the manner of introduction of WPBA.
Table 2. Trainees’ attitudes and perceptions to WPBA, its effects, practicalities of assessments and the Royal College of Psychiatrists’ assessment toolsa
| Strongly disagree | Disagree | No opinion | Agree | Strongly agree | |
|---|---|---|---|---|---|
| % (n) | |||||
| Workplace-based assessment: | |||||
| is backed by good evidenceb | 17 (13) | 42 (33) | 30 (24) | 11 (9) | 0 (0) |
| is validb | 16 (13) | 35 (28) | 26 (21) | 23 (18) | 0 (0) |
| is reliable | 24 (19) | 44 (36) | 17 (14) | 14 (11) | 1 (1) |
| is independent of the assessor's personal preferences | 28 (23) | 53 (43) | 9 (7) | 5 (4) | 5 (4) |
| has appropriately been made compulsory | 28 (23) | 41 (33) | 17 (14) | 12 (10) | 1 (1) |
| has been introduced in a well thought-out manner | 31 (25) | 52 (42) | 11 (9) | 5 (4) | 1 (1) |
| has been accompanied by sufficient and relevant information and guidance | 30 (24) | 38 (31) | 11 (9) | 19 (15) | 3 (2) |
| has been accompanied by sufficient trainingc | 36 (29) | 39 (31) | 18 (14) | 5 (4) | 3 (2) |
| Workplace-based assessment has improved: | |||||
| supervision | 14 (11) | 40 (32) | 16 (13) | 27 (22) | 4 (3) |
| training | 19 (15) | 42 (34) | 21 (17) | 16 (13) | 3 (2) |
| clinical practice | 17 (14) | 44 (36) | 20 (16) | 16 (13) | 3 (2) |
| confidence | 21 (17) | 41 (33) | 17 (14) | 19 (15) | 3 (2) |
| Assessments: | |||||
| are easy to organise | 28 (23) | 48 (39) | 11 (9) | 12 (10) | 0 (0) |
| have no impact on time available for clinical duties | 21 (17) | 49 (40) | 15 (12) | 14 (11) | 1 (1) |
| are facilitated by easy access to computers | 38 (29) | 41 (33) | 12 (10) | 11 (9) | 0 (0) |
| receive the necessary support from supervisors and colleagues | 14 (11) | 30 (24) | 21 (17) | 33 (27) | 3 (2) |
| Assessment tools adopted by the College:b | |||||
| are easy to use | 14 (11) | 32 (25) | 20 (16) | 32 (25) | 3 (2) |
| are relevant to psychiatry | 5 (4) | 27 (21) | 24 (19) | 43 (34) | 1 (1) |
| facilitate training well | 14 (11) | 44 (35) | 28 (22) | 13 (10) | 1 (1) |
| Overall, workplace-based assessment in its current form:c | |||||
| is being used appropriately | 14 (11) | 40 (32) | 28 (22) | 18 (14) | 1 (1) |
| is acceptable to you as a trainee | 19 (15) | 35 (28) | 24 (19) | 21 (17) | 1 (1) |
| accurately reflects trainees’ progress | 25 (20) | 43 (34) | 18 (14) | 13 (10) | 3 (2) |
| is a better system than the previous one | 24 (19) | 35 (28) | 24 (19) | 15 (12) | 3 (2) |
| is the way forward and should be retained | 30 (24) | 28 (22) | 29 (23) | 11 (9) | 3 (2) |
The majority of trainees feel that WPBA has had no real beneficial effects on supervision, training, clinical practice and confidence.
Over 70% of trainees feel that there are difficulties organising assessments and that assessments have a negative impact upon time for clinical duties, 44% of trainees harbour concerns about lack of support from supervisors and colleagues, and 79% have concerns about the lack of access to computer facilities in the context of assessments.
Assessment tools
Varying concerns were raised about the College's assessment tools: 58% of trainees feel that they do not facilitate training, 46% that they are not easy to use, and 32% that they are not relevant to psychiatry.
Overall perceptions about WPBA in its current form
Between half and two-thirds of trainees feel that WPBA in its current form is unacceptable to them as trainees, does not accurately reflect their progress, is no better than the system in use before the MMC, and is not the way forward and therefore should not be retained. Fewer than one in four trainees supported positive statements in these contexts.
Trainees’ perceptions of their assessors
For the purpose of this survey, ‘acceptable’, ‘good’ and ‘very good’ were considered as adequate standards.
Consultant assessors
Educational (consultant) supervisors were rated well in terms of ability to assess accurately and impartially (93% of trainees), willingness to complete assessments (91%), ability to provide constructive feedback (89%), computer literacy (88%), and availability (85%) (Table 3). They were rated moderately on knowledge of WPBA (66%) and understanding of the online system (70%).
Table 3. Trainees’ perceptions of their assessors
| Consultant assessors (N = 80) | Non-consultant assessors, (N = 77) | Non-doctor assessors, (N = 81) | |
|---|---|---|---|
| % (n) | |||
| Availability | 69 (85)a | 51 (66) | 50 (62) |
| Willingness to complete assessments | 73 (91) | 58 (75) | 58 (72) |
| Knowledge of workplace-based assessments | 53 (66) | 44 (57) | 5 (6) |
| Computer literacy | 70 (88) | 69 (90) | 44 (54) |
| Understanding of the online system | 56 (70) | 49 (64) | 15 (19) |
| Ability to assess accurately and impartially | 74 (93) | 57 (75)b | 50 (62) |
| Ability to provide constructive feedback | 71 (89) | 56 (73) | 45 (56) |
Non-consultant medical assessors
Non-consultant medical assessors were rated well on computer literacy (90%), and moderately well in terms of willingness to complete assessments (75%), ability to assess accurately and impartially (75%), ability to provide constructive feedback (73%), availability (66%), understanding of the online system (64%) and knowledge of WPBA (57%). They scored consistently lower than consultants on every parameter except computer literacy.
Non-medical assessors
Non-medical assessors scored lower than medical assessors on every parameter. They were rated moderately in terms of willingness to complete assessments (72%), availability (62%), ability to assess accurately and impartially (62%), ability to provide constructive feedback (56%) and computer literacy (54%). Scores were less impressive in terms of understanding of the online system (19%) and knowledge of WPBA (6%).
Trainees’ perceptions about recording assessments
Once again, ‘acceptable’, ‘good’ and ‘very good’ were considered adequate standards. Overall, trainees did not rate the online system very highly (Table 4). The percentages of trainees who found it unacceptable were: 78% in terms of ease of use, 75% in terms of reliability, 70% in terms of time consumption, and 96 % in terms of the appropriateness of the fee that trainees will have to pay. Overall, from the trainees’ perspective, recording assessments online would not render the process of WPBA any easier compared with a paper-based system.
Table 4. Trainees’ perception of the online system of recording assessments (n = 78)
| Very poor | Poor | Acceptable | Good | Very good | |
|---|---|---|---|---|---|
| % (n) | |||||
| Ease of use | 51 (40) | 27 (21) | 15 (12) | 5 (4) | 1 (1) |
| Reliability | 44 (34) | 31 (24) | 19 (15) | 5 (4) | 1 (1) |
| Time consumption | 35 (27) | 35 (27) | 21 (16) | 9 (7) | 1 (1) |
| Appropriateness of the fee | 68 (53) | 28 (22) | 3 (2) | 0 (0) | 1 (1) |
The majority of trainees (51%) would prefer assessments to be recorded on paper (Table 5), although 38% would be happy for a combination system (i.e. largely online, provided assessments could be done on paper if an assessor could not be brought to a computer). Only 6% of individuals would prefer an online system (Table 6).
Table 5. Trainees’ perception of recording of assessments online in comparison with on paper (n = 80)
| Strongly disagree | Disagree | No opinion | Agree | Strongly agree | |
|---|---|---|---|---|---|
| % (n) | |||||
| Will be easier | 40 (32) | 39 (31) | 8 (6) | 11 (9) | 3 (2) |
| Will promote better compliance | 25 (20) | 44 (35) | 11 (9) | 16 (13) | 4 (3) |
| Will be more reliable | 26 (21) | 40 (32) | 18 (14) | 13 (10) | 4 (3) |
| Will consume less time | 34 (27) | 45 (36) | 13 (10) | 6 (5) | 3 (2) |
| Will be more cost-effective | 30 (24) | 35 (28) | 16 (13) | 15 (12) | 4 (3) |
| Will be easier for assessors to complete forms | 41 (33) | 45 (36) | 10 (8) | 0 (0) | 4 (3) |
Table 6. Trainees’ preference with regard to a system for recording assessments
| % (n) | |
|---|---|
| On paper | 51 (41) |
| Online | 6 (5) |
| Either (both are equally good) | 5 (4) |
| Combination (largely online, but on paper if an assessor cannot be brought to a computer) | 38 (31) |
Trainees’ concerns about the WPBA
The survey also incorporated the option for respondents to express in free text their views on WPBA. The most common comments and concerns were, in summary:
-
• WPBA proving to be a ‘tick-box’ exercise to fulfil annual review of competence progress requirements with few real training benefits, particularly for post-membership trainees
-
• lack of established standards for various training grades (ST1-6)
-
• significant difficulties organising and completing assessments (particularly online)
-
• lack of enthusiasm from assessors
-
• poor quality of trainers
-
• time spent performing WPBAs detracts from clinical work
-
• inconsistencies in assessments - high degrees of subjectivity, assessor bias
-
• patchy and inadequate training to assessors
-
• unreliable and unpopular online system.
Recommendations by trainees
Respondents were also invited to offer their recommendations with regard to remedial measures that might improve the WPBA system. The most common recommendations included:
-
• targeted training for assessors, particularly non-medical, will be essential (this was a consistent theme);
-
• due to current unreliability the online system needs major overhauling or should be abolished - if retained, there should be easy access to computers and no fee for the online system;
-
• methods should be devised to standardise assessments between assessors;
-
• there should be an option of recording WPBA on paper;
-
• skills that are appropriate for assessment by non-doctors should be reviewed and defined, since such assessments are relevant to some skills but not others;
-
• the assessment tools should be rendered relevant to the individual specialty and should be more flexible;
-
• the responsibility for completing assessments should not rest solely with the trainee;
-
• trainees should be given protected time to organise WPBA;
-
• the number of assessments should be reduced and the relevance of each one should be improved: the results of assessments should be interpreted and summarised in a meaningful manner;
-
• WPBA should be tailored to the level of the trainee (e.g. higher trainees should be assessed by suitable and appropriate members of staff).
Discussion
The educational principle underpinning WPBA appears to be sound, with immense potential if used appropriately, and it is striking that a large proportion (43%) of psychiatric trainees in Wales do believe that WPBA was intended to improve training. The new framework has, however, engendered overwhelmingly negative attitudes, partly due to a common perception (41%) that it is politically motivated with its rationale and purpose unclear, but perhaps largely due to the manner in which it has been introduced (83% feel that it has not been introduced in a well thought-out manner). Moreover, WPBA is widely perceived as being conceptually flawed, based on scant evidence and of dubious validity as an assessment tool.
Other changes introduced with WPBA
It is of relevance that this completely novel assessment system was introduced simultaneously with radical changes to medical training (MMC), medical recruitment (Medical Training Application Service, MTAS) and the core curriculum. The resultant confusion and uncertainty with regard to both the present and the future has had a significant impact on trainees’ confidence and morale. Reference Tooke3,Reference Lydall, Malik and Bhugra4 Although any change tends to provoke resistance, changes imposed under such circumstances further heighten reluctance to participate. Moreover, rather than a phased and gradual introduction, WPBA was made mandatory from the outset, for all specialty trainees appointed under MMC. Changes were rushed through, with little time for comprehension, acceptance and adaptation by both trainees and trainers.
Time constraints and other pressures
It is also relevant that the entire responsibility for maintaining a performance-based portfolio has been placed on the trainee. This combines with time constraints (and perhaps lack of faith in a new unproven process) on the part of assessors in creating significant difficulties for the trainee. To further complicate the picture, it is relevant that non-fulfilment of the new requirements could result in trainees being failed at their annual appraisals, with potential denial of progress up the training ladder. The process is therefore at considerable risk of degenerating into a tick-box/paper-pushing exercise, which nevertheless, by its very nature, is likely to take priority over the acquisition of essential clinical experience.
Apart from this, WPBA is intrinsically time and resource intensive. Increased levels of supervision and assessment of trainees, integral to the organisation and completion of the new performance-based portfolio, will undoubtedly detract from time available for the performance of clinical duties, and in combination with the recent reductions in junior doctors’ working hours could paradoxically run counter to the stated intention of improving patient care. Surprisingly, despite the extra work involved for both trainers and trainees, time implications have received little official consideration, there having been no formal allocation of time for WPBA in either consultant contracts or trainee timetables. Reference Grant5
Problems with HcAT
A further source of frustration to trainees involved the introduction of an online system (Healthcare Assessment and Training, HcAT), with a view to permanently storing assessments and generating summary reports from completed assessments. 6 It was intended that HcAT would be rendered compulsory and also that it would constitute the only acceptable form of evidence of training. Apart from the fact that the benefits of online documentation are debatable, the resultant difficulties faced by both trainees and assessors, in terms of awareness of the new process, computer literacy, access to computers and indeed time were immense.
The majority of the trainees found the online system of recording assessments unacceptable in terms of time consumption (70%), reliability (75%) and ease of use (78%) (Table 4). The HcAT system was plagued by numerous difficulties, so much so that it was abandoned by the Royal College of Psychiatrists in favour of ‘Assessments Online’, an updated system which appears to have addressed some of the technical problems associated with HcAT, although some difficulties remain. Any online system, however, is intrinsically prone to user factors, not the least of which involves adherence from assessors (particularly non-medical).
The fee for the online assessment system also carries potential for further resentment (for 96% of trainees this is unacceptable), since registration with the College is now mandatory for training to be recognised. 7
Comparison with the previous appraisal system
A small but significant proportion of trainees (17%) had no experience under the previous training system and their views about WPBA may therefore be overly optimistic or pessimistic. Nevertheless, given the overall current opinion expressed by trainees, it is evident that much work remains for WPBA to effectively address the deficiencies of the previous RITA appraisal system that were identified by PMETB: regional variability, lack of quality assurance and governance mechanisms, a perception by doctors that it was a bureaucratic and form-filling exercise, and paucity of time and resources allocated for training. 8 Advocates of WPBA may argue that the current difficulties are merely ‘teething problems’ inherent to any new system which will resolve in time. Although this may evoke instant scepticism, if there is indeed substance to this point of view, it is certainly unacceptable for current trainees to become casualties of yet another new initiative introduced, from the trainees’ perspective, without adequate thought and consideration from policy makers.
The way forward
It would appear that a great deal of essential work remains with regard to validating the assessment tools adopted, determining the optimal number of assessments required, exploring the effects of employing non-medical assessors to assess senior trainees, establish reliability, validity and repeatability of assessments, and correlating WPBA scores with the achievement of various competencies. Reference Searle9 It is tempting to speculate whether WPBA would have been met with a more cordial reception had these factors been addressed before its implementation. It would intuitively appear that many of the current problems might have been avoided had WPBA been introduced in a phased and structured manner, with preliminary address of issues around training of assessors and due consideration given to feedback from both trainees and trainers.
A standardised, transparent and reproducible system of appraisal carries obvious advantages in terms of affording robust assessment and certification of trainees’ competencies. Well-informed and competent assessors (which may include consultants, other senior doctors, psychologists, nurses and social workers) are essential for such a system to function effectively. Standardisation, however, mandates consistency, which in turn requires structured training and quality assurance. Training has been perceived as patchy at best, as evidenced by the poor scores allocated to assessors, particularly those from non-medical backgrounds, by an overwhelming majority of trainees, and yet there is no process for quality assurance. Perhaps worryingly, non-medical assessors were rated very poorly on the three most important parameters: understanding of WPBA (6%), the ability to assess accurately and impartially (62%), and the ability to provide constructive feedback (56%). This carries serious implications in that, for instance, an assessment by an inadequately trained assessor carries a higher than acceptable risk of inaccurately reflecting a trainee's ability or performance, with obvious potential to inappropriately boost or damage their confidence with attendant implications. It is interesting that even consultants, the group rated most highly by trainees, scored poorly on knowledge of WPBA and understanding of the online system. Structured and specific training of all assessors is therefore a fundamental requirement in order to minimise interpersonal variability (currently inherent to this system) and in doing so, to maximise the objectivity of assessments. Training of assessors is also relevant to equip them (if required) to deliver negative feedback to a trainee in a constructive and supportive manner, to ensure that assessment becomes a productive learning exercise. As a corollary, it is equally important that trainees too are educated about WPBA, so as to preclude the development of resentment and demoralisation in response to a less than flattering assessment. The confidence of all stakeholders would also be enhanced by the development of an effective quality assurance process.
Competency-based assessment is new to psychiatry and represents uncharted territory. Like any system of training, it carries both strengths and limitations. Needless to say, it is imperative that robust measures be employed to continuously build on its strengths, remedy its intrinsic limitations and tailor it to the idiosyncrasies of psychiatric training, while facilitating widespread understanding, cooperation and uptake, in order to facilitate the development of a new generation of comprehensively trained doctors.
Study strengths and limitations
Despite the intrinsic limitations of a cross-sectional survey, the high response rate indicates that the results are representative of the views of psychiatric trainees in Wales; indeed, this high response rate, unusual for this type of study, perhaps indicates the strength of feeling among trainees on this subject. This survey was conducted approximately a year after the introduction of WPBA and therefore involved respondents who had acquired first-hand practical experience with this system of assessment.
One possible limitation of a questionnaire-based survey involves the narrow scope and possible inadequacy of the spectrum of selectable responses for any given question with regard to accurately expressing the views of participants (a respondent might, for instance, select the ‘no opinion’ box for lack of one that best expresses their views). This survey sought to address this by affording responders the option to state, in free text, any further views they wished to express over and above the areas addressed specifically by the questionnaire. There again, this study design affords the advantage of a reduced level of ‘observer bias’ since the anonymity enjoyed by the respondents removes the constraints with regard to expressing their views freely.
Declaration of interest
None.
Open access
eLetters
No eLetters have been published for this article.