Can trainees contribute to recruitment into specialty training?

Abstract

Aims and Methods

Following the 2007 Modernising Medical Careers process, an online survey of applicants for psychiatry training posts in Scotland was undertaken. Views on short-listing, interview, communication and future recruitment were sought, qualitative and quantitative analyses being employed.

Results

Of 200 doctors, 115 responded, representing all regions and grades. Over 75% (n=84) expressed lack of confidence in short-listing. There was no significant difference between responses of successful and unsuccessful candidates (χ²=3.66, d.f.=2, P > 0.05). A total of 45% perceived interviews as invalid (n=55). About two-thirds (66%, n=69) felt communication was poor. Constructive suggestions for 2008 were made.

Clinical Implications

Although highlighting inadequacies in the 2007 process, trainees offered valid suggestions for future recruitment into specialty training, information being obtained quickly and efficiently.

There has been considerable debate about inadequacies in the Modernising Medical Careers (MMC) process across all medical specialties over the last 2 years. Trainees complain that the scheme has had adverse effects on their health and quality of work (Lydall et al, 2007) and point to problems in the Medical Training Application Service (Whelan et al, 2007). In Scotland, the process differed significantly from the rest of the UK, with single-stage recruitment, all applicants being interviewed on one occasion therefore, preceding the Douglas Review (Tooke, 2008) which heavily criticised the process. Changes have been proposed for future recruitment, including the comprehensive Tooke report published in January 2008 (Tooke, 2008).

In view of psychiatric trainees’ concerns about recruitment, trainee representatives of the Specialty Board for Psychiatry in Scotland and the Psychiatric Trainees Committee of the Royal College of Psychiatrists undertook an online survey of all current psychiatric trainees in Scotland. The survey reviewed the 2007 recruitment process and provided informed opinion for the 2008 recruitment.

Method

Themes for this survey were proposed by members of the Transitional Board for Psychiatry in Scotland; questions were identified by a representative sample of psychiatric trainees (from different regions and at different training levels) in an online pilot survey. Subsequently, all Scottish trainees received the web-link to the survey through the regional deanery leads for all four Scottish regions. The survey was available online for 2 weeks. The questions addressed both the 2007 and the 2008 recruitment process. We asked about short-listing, interview, communication and feedback, and suggestions for future selection. The survey contained both open- and closed-ended questions (using a Likert scale for both).

Statistical analysis

Responses to closed-ended questions were analysed with descriptive and inferential statistical methods, using online software (www.surveymonkey.com). Open-ended questions were analysed using a qualitative content analysis method, which has been defined as ‘a research technique for the objective, systematic and quantitative description of the manifest content of communication’ (Berelson, 1971). One of us (P.S.) identified themes, which were then explored by two coders independently (S.J. and T.H.). Any discrepancies in the identified themes were resolved through discussion between S.J. and T.H.

Results

Details of results are given as percentages of those who answered that specific question in the survey.

2007 recruitment

Of all trainees in Scotland (n=200), we obtained 123 responses (115 trainees and 8 staff grade doctors) and 113 respondents (94%) fully completed the survey. Eight respondents (not specifically staff grade doctors) did not compete for training posts in 2007. The participants’ grades and regions were representative of all psychiatry trainees in Scotland: there were 20 specialist trainees in year 1, 26 trainees in year 2, 36 in year 3, 8 in year 4 and 20 doctors on fixed-term specialty training appointment. The trainees were from all Scottish regions: 60 were from the west, 36 from the south east, 20 from the north and 7 from the east. Of those completing the survey, 68 (56%) were awarded a post of their choice in August 2007.

Short-listing

Over two-thirds of participants (n=84, 76%) expressed lack of confidence in the short-listing process (Box 1). They thought the form was either too subjective (n=79, 74%) or too objective (n=13, 12%); for 15 participants (14%) the form was just right. There was no statistically significant difference between the responses of successful and unsuccessful candidates (χ²=3.66, d.f.=2, P>0.05).

Participants criticised the validity of the questions’ content. One of them commented that ‘… subjective questions about “stressful situations” and the like bear no resemblance to clinical competence’. Furthermore, responses indicated concerns whether and to what extent were the scenario-based questions reliable.

Interview

Almost half the participants (n=55, 45%) thought the interviews invalid; for many (n=40, 37%), they were either too short or too long. There were statistically significant differences between the views of successful and unsuccessful candidates – the latter expressed more critical views regarding the interviews’ validity (χ²=21.8, d.f.=2, P<0.05) and duration (χ²=6.5, d.f.=2, P<0.05).

Communication and feedback

Approximately two-thirds of the participants (n=69, 66%) considered the feedback they received after the process either poor or extremely poor. The majority of trainees (n=92, 87%) approved of facilitating feedback via email. However, only half of the participants (n=55, 51%) thought that email communication was appropriate for setting interview dates and only 42% (n=45) considered it appropriate for appointment offers.

Box 1. Main themes in the Scottish trainees’ survey

1. • Lack of confidence in the Medical Training Application Service short-listing process; responses did not differ between successful and unsuccessful candidates

2. • Most of the participants (66%) thought communication and feedback were poor or extremely poor

3. • Constructive suggestions were offered

   These views informed recruitment in 2008.

In relation to overall communication, approximately two-thirds of participants (n=74, 68%) did not receive any feedback. Of those who did, only 56% (n=23) found this helpful. Generally, the response pattern did not differ significantly between successful and unsuccessful candidates.

Recruitment 2008

We received a high number of comments about future recruitment. For short-listing, the majority of participants (n=82, 78%) favoured a combination of structured CV and application form. The following were the most common suggestions:

1. • elimination of ‘describe a time…’ narrative questions; instead, greater emphasis on objective evidence

2. • longer word limits for descriptive sections

3. • previous clinical experience to be taken into consideration

4. • extending the time limit for the application process

5. • use of the form only to check eligibility

6. • establishing demonstrable competencies, based on clinical practice.

A large number of participants thought that their CV and portfolio should be available at interview; however, that was not the case for trainees in their third (n=106, 86%) and fourth year (n=113, 92%). Most of the respondents (n=103, 93%) agreed that references should be available to the interview panel and the majority (n=77, 63%) suggested that structured references should be made available at interview.

Discussion

This is the first survey to highlight the main stakeholders’ views on recruitment into specialty training in psychiatry. It has had a direct impact on future recruitment in Scotland, as the results were presented to the recruitment group of the Specialty Board for Psychiatry in Scotland.

Dissatisfaction with short-listing is in keeping with recent criticism of Medical Training Application Service (Whelan et al, 2007; Tooke, 2008). More weight is given to this finding when one acknowledges that this opinion was shared by both successful and unsuccessful candidates. In Scotland, all applicants were interviewed and all posts allocated entirely on the basis of the interview scoring. This system was criticised nationally and led to changes in the way subsequent interviews were organised in England and Wales (Tooke, 2008). In Scotland, the format of the interview was decided on by the Scottish Executive (government), rather than by transitional boards. The decision to interview all applicants put Scottish trainees at a major disadvantage. It is no surprise that successful candidates thought the interview process valid as opposed to unsuccessful candidates, as this was the sole means of assessment. It would have been worthwhile compiling qualitative feedback on this aspect of the process in our survey.

One of the clearest findings of the survey was the perceived lack of adequate feedback processes during the 2007 recruitment in Scotland, as was also highlighted in the Tooke report (Tooke, 2008).

Strengths of this survey include the high response rate from trainees in Scotland, reflecting trainees’ interest in providing feedback to the Specialty Board and their motivation to contribute to future recruitment.

Trainees who had been successful in the application process also shared their views, therefore our findings do not merely represent the misgivings of disappointed applicants. An independent observer (T.H.) who was not involved in the MMC selection process contributed to the analysis. Two of us (S.J. and M.E.) applied for posts in 2007. Another two (S.J. and P.S.) had extensive consultation with Scottish trainees throughout the MMC process and received substantial informal feedback, which could have influenced the survey questions. It is heartening that the Specialty Board for Psychiatry in Scotland considered the results of our survey for 2008 recruitment. In light of this, a further study should be undertaken to ascertain whether trainees felt their feedback had any impact on this year's recruitment process.

Conclusions

Despite being subjected to glaring deficiencies inherent within the MMC process in Scotland, psychiatry trainees offered constructive and unbiased suggestions to inform decision-making for future recruitment. Their views were obtained quickly and efficiently, using an online survey. In keeping with recent recommendations, future quality assurance and governance structures should incorporate transparent, formalised feedback from trainees to ensure future recruitment processes do not engender the degree of distress seen in 2007.

Declaration of interest

S.J. was Honorary Secretary of the Psychiatric Trainees Committee from 2005 to 2007 and has been trainee representative of the Specialty Board for Psychiatry in Scotland since its inception in 2006. P.S. has been a member of the Psychiatric Trainees Committee from 2006 and a member of the Specialty Board for Psychiatry in Scotland from 2007.