prepOSCE: A Virtual, Scalable Solution to Prepare Residents for Their OSCE Examination

Abstract:

The Royal College Comprehensive Objective Examination in Neurology provides certification for Canadian neurologists and consists of a written examination and the Observed Structured Clinical Encounter (OSCE). The OSCE portion of the certification involves residents visiting several patient stations where they address case scenarios with an examiner. Unfortunately, residents lack exam preparation time due to demanding work hours. In response to resident needs, we created a novel, virtual preparation OSCE program – “prepOSCE” – and evaluated its efficacy. The prepOSCE program employed a proprietary virtual solution from CTC Communications Corp. Ten virtual sessions accommodated 70 residents totally. Seven Canadian physicians and two co-chairs created case scenarios for the stations. On session day, seven residents arrived in a virtual plenary room for briefing followed by assignment to a station by CTC. Residents then moved virtually through prepOSCE stations a different examiner and case scenario in each. Following their session, residents and evaluators were surveyed to capture experiences. The average program rating was 4.22 out of 5 (n = 36 residents of 70 residents who participated in the program) and 4.35 (n = 17 evaluators). Ninety-two percent of residents agreed or strongly agreed that they would recommend this program to their peers; they would like prepOSCE to continue next year; and the program was relevant and added value to their studies. The positive feedback received from prepOSCE participants indicates there is a need for a program like prepOSCE. This model has potential for expansion and it is hoped that specialties outside of neurology could benefit from a similar program.

Résumé :

préECOS : solution virtuelle et échelonnable de préparation des résidents à l’examen ECOS.

L’examen objectif intégré du Collège royal en neurologie conduit à la reconnaissance professionnelle des neurologues au Canada, et il consiste en un examen écrit et en l’examen clinique objectif structuré (ECOS). Pour cette dernière partie de l’évaluation, les résidents et résidentes doivent aller dans plusieurs postes de patients et évaluer différents cas, en la présence d’un examinateur. Malheureusement, les résidents manquent de temps pour se préparer convenablement à l’examen, en raison des longues heures de travail. Alors, afin de répondre aux besoins des résidents, nous avons élaboré un tout nouveau programme de préparation virtuelle à l’ECOS, appelé préECOS, et en avons évalué l’efficacité. Le programme préECOS reposait sur une solution virtuelle, propriétaire de CTC Communications Corp. Au total, 70 résidents ont profité de 10 séances virtuelles. Sept médecins canadiens et deux coprésidents ont conçu des études de cas pour les postes de consultation. Durant les jours de séance, les résidents, au nombre de sept, participaient en séance plénière, dans une salle virtuelle, à une réunion préparatoire, puis étaient dirigés vers un des postes de consultation par l’équipe de CTC. Les résidents passaient d’un poste de préECOS à un autre, en mode virtuel, chaque fois accompagnés d’un examinateur différent et soumis à un nouveau cas. À la fin des séances, les résidents et les évaluateurs répondaient à un questionnaire d’appréciation de leur expérience. La note moyenne s’élevait à 4,22 sur 5 pour les résidents (n = 36  sur 70 participants) et à 4,35 pour les évaluateurs (n = 17). Ainsi, 92 % des résidents se sont dits d’accord ou fortement d’accord avec les énoncés suivants : ils recommanderaient le programme à leurs pairs; ils souhaiteraient que le programme préECOS se poursuive l’année suivante; ils considéraient que le programme était pertinent et qu’il ajoutait de la valeur à leurs études. La rétroaction favorable des participants au programme préECOS est révélatrice de la nécessité de ce type de formation. Le modèle offre un potentiel d’expansion et il est à souhaiter que les résidents dans d’autres spécialités que la neurologie puissent profiter de programmes similaires.

The Royal College Comprehensive Objective Examination in Neurology (Adult and Pediatric) provides certification for practising neurologists in Canada and consists of two parts: a written examination and the Observed Structured Clinical Encounter (OSCE) component. Whether done live or virtually, the OSCE examination involves residents visiting a number of separate “stations” where they are asked to address case scenarios with an examiner.

Unfortunately, neurology residents do not have sufficient time to prepare thoroughly for these exams: more than half of Canadian residents work more than 60 hours per week and almost 30% work at least 70 hours per week. ¹ In addition, as far as could be determined, there is currently no national standard for neurology review programs to help residents prepare for the OSCE exams. It appears that Canadian residents would benefit from a learning program that would fit into their busy schedules and help prepare them for successful completion of their exams.

In response to the needs of Canadian neurology residents, we created a virtual preparation program for the OSCE exam, entitled “prepOSCE,” and evaluated it for efficacy in preparing Canadian neurology residents for their OSCE.

A number of universities across the world recently carried out their medical, dentistry, or nursing OSCE exams in a virtual manner due to the restrictions resulting from the COVID-19 pandemic. ^2–7 Notably, the Royal College opted to deliver virtual OSCEs for both the 2021 and 2022 sessions. Some educational programs have included virtual components for the purpose of teaching residents how to effectively provide virtual care. ^7–10 However, there is only one published account of virtual preparation OSCE exams in the literature with the purpose of preparing residents for their eventual OSCE exams. ¹¹ That program involved 14 residents, while prepOSCE involved five times as many. Another medical school instituted a virtual mini-course in clinical reasoning that included an OSCE practice examination, but this was conducted in group sessions rather than by individuals. ¹² Thus, it appears that prepOSCE, if only by numbers involved, is currently unique in the literature.

The prepOSCE program is also unique for another reason. Most practised OSCEs, whether in-person or virtual, are conducted within the confines of a single university. As a result, residents have no opportunity to practise with evaluators (examiners) not known to them. This may be mitigated through collaboration with neighbouring universities, but there are limitations to this. Because of its cross-Canada presence, prepOSCE makes it possible for residents to practise with evaluators they may never have previously met.

The goals of the prepOSCE program were to increase (1) awareness of and comfort with Royal College Examination processes and (2) neurology-specific knowledge, skills, and competency among near-final- and final-year Canadian neurology residents.

The prepOSCE program used a proprietary solution from CTC Communications Corp. that combines technology, operations, and virtual meeting capabilities. Ten half-day sessions were set up to allow 70 residents to participate (7 per session). A preparatory survey was done, which determined that most residents felt comfortable or very comfortable using virtual platforms such as Zoom™ and Skype™.

Seven Canadian physicians with academic affiliations created case scenarios to be presented at each station and two co-chairs were responsible for determining the case scenario topics and content review. Each case scenario creator was assigned a neurology topic based on their expressed interest and/or expertise, and these assignments were approved by the co-chairs. Case scenario creators were provided with a prepOSCE case scenario template that was developed by CTC Communications Corp. to closely mirror the components of OSCE case scenarios. Draft cases were reviewed by the co-chair before finalization.

The topics for the seven case scenarios were as follows:

1. 1. Neuromuscular disorders

2. 2. Stroke

3. 3. Pediatric neurology

4. 4. Communication (with a focus on epilepsy)

5. 5. MS/differential diagnosis

6. 6. Movement disorders

7. 7. Headache/neuroophthalmology.

Invitations to residents were disseminated by the Canadian Neurological Sciences Federations (CNSF). Residents could register for a prepOSCE event at neuro.preposce.ca up to 24 hours before a scheduled event.

Evaluators were recruited by the CNSF and CTC and availability confirmed before prepOSCE dates were listed on the registration website. Evaluators were offered one of two training sessions to help them prepare for prepOSCE and understand their roles before, during, and after the event. The training sessions provided evaluators with standardized instruction on how to lead and evaluate the stations. During the training sessions, evaluators received guidance on the standardized evaluation forms designated for each station. They also completed a dry run and a technology check. Evaluators could, and did, participate in more than one prepOSCE event.

Registered evaluators were assigned to stations and sessions with an effort to avoid overlap between residents and evaluators from the same institution.

Each prepOSCE event was attended by 3 CTC personnel, who acted as facilitator (general speaker that opened and closed the event and ran pre-meeting technology checks), coordinator (managed resident movement through stations and was primary time keeper), and IT support (managed all technology-related issues for both residents and evaluators).

On their session day, residents arrived in a virtual plenary room for a briefing session, after which they were each assigned to a station. They then went through virtual meeting breakout rooms that mimicked OSCE stations, each with an examiner. Upon arrival in the virtual breakout room the resident was presented with doorway information that outlined the case scenario to be completed for that station (Figure 1). As in the Royal College OSCE examination, the residents moved from station to station and were presented with a different case scenario in each (Figure 2). Live tech support was available to both residents and evaluators throughout the session.

Figure 1: A screenshot of the doorway information to a sample case scenario in the format presented to residents.

Figure 2: The timing and the roles and responsibilities of the residents and evaluators throughout a prepOSCE event.

Evaluation mechanisms (surveys) for both residents and evaluators were designed to capture and analyze their experiences. Evaluators at each station filled in a unique evaluation profile and subtotals for each section that were specific to the station (Figure 3).

Figure 3: A screenshot of a blank evaluation form template (to be completed based on the station).

Seventy-nine students registered for the program, of whom 70 were residents (CNSF junior members); they came from 16 universities in eight provinces. Twenty-two physicians registered to be evaluators; they came from eight universities in five provinces. The residents’ mean scores for the seven stations are shown in Table 1.

Table 1: Completion rates and mean scores for each station

The average rating for the program overall was 4.22 out of 5 on the part of residents (n = 36) and 4.35 on the part of evaluators (n = 17).

Ninety-two percent of residents that participated in the post-program survey said that they agreed or strongly agreed that they would recommend this program to their peers; that they would like prepOSCE to continue next year; and that the program was relevant and added value to their studies (Figure 4).

Figure 4: Results of an evaluation question for participants.

Interviews done after both the prepOSCE and the Royal College OSCE were over produced a number of valuable comments from the residents. Examples included:

• “The quality of the cases was good and quite representative of the real OSCE.”

• “This OSCE was smoother with less IT issues than my actual Royal College OSCE.”

• “Each station was nicely compartmentalized into sections.”

• “Having evaluators we donʼt know is a huge advantage – it gives a wider breadth of exposure.”

• “There are never enough practice tools, so please continue this one.”

• “The interface was different from the Royal College interface, but quite comparable.”

• “As additional resources on the prepOSCE website, a repository of previous stations would be helpful.”

• “I didnʼt appreciate the examiner acting as a patient.”

• “The examiner couldnʼt see what the resident could see.”

• “It was a bit difficult to keep to the time – I felt rushed at a few stations. The examiners did not have time to give feedback.”

• “The personal preferences of the examiners affected the cases – one examiner was not comfortable sharing the sexual history of a 17-year-old female.”

• “Some of the examiner feedback was inconsistent with others’ feedback.”

A few additional themes emerged: (1) a station could contain multiple case scenarios, put forth in quick succession; (2) the level of difficulty of the case scenarios was similar in prepOSCE and the Royal College OSCE; and (3) greater standardization of the prepOSCE examiners would be valuable.

All the evaluators agreed or strongly agreed that the program was well organized and well executed; that the program’s delivery was innovative; that they would recommend this program to residents; and that they would participate in a prepOSCE conducted next year (Figure 5). Examples of responses to the question “What part of prepOSCE was most valuable to you?” included, “The opportunity to help residents through a stressful time” and “Face-to-face meetings with residents without the constraints of travel.”

Figure 5: Results of an evaluation question for evaluators.

Interviews done after the prepOSCE event brought out a number of useful comments from the evaluators:

• “More practice for the Royal College exams is always better.”

• “Communication with the CTC team was fine. There was one issue but it was tackled smoothly.”

• “Support during the session was excellent – better than at the actual OSCE.”

• “Guidance on how to monitor time would have been helpful.”

• “Use of a standardized patient is hard to do and I’m not sure what it necessarily adds.”

• “A dialogue between the case creator and the examiner would have helped.”

• “Usually in an OSCE, there’s a dry run with the examiner to get a feel for the case. I would have appreciated having that here – expectations werenʼt clear.”

• “The standards for judging were not sufficiently clear.”

Additional themes emerged: (1) online or paper marking sheets would be preferred over the ones used; (2) the case creation templates would benefit from being more flexible; (3) residents benefit from having examiners they do not know; and (4) greater standardization of the prepOSCE examiners would be valuable.

It is apparent that virtual OSCE preparation examinations are feasible and scalable. The prepOSCE approach offers a functional solution that combines the use of technology, operations, and virtual meeting capabilities. Although the COVID-19 pandemic has accelerated the development of virtual meetings, a program like prepOSCE has applications beyond the end of the pandemic since it offers a cost-effective, scalable model of an extended preparatory session to residents.

The model can self-evidently be extended beyond neurology. In 2019, there were more than 91,000 physicians in Canada ¹³ and it has been estimated that there is one resident for every five Canadian physicians. ¹⁴ This suggests that there are more than 18,000 Canadian residents who could theoretically benefit from an exam preparation program such as prepOSCE. Psychiatry, internal medicine, pediatrics, and cardiology are some of the major therapeutic areas to which this program could be extended. Future programs should seek to increase resident participation in post-program surveys to ensure the feedback gathered represents the experience of all the participating residents. Simple incentives, such as pairing release of evaluation feedback with participation in the post-meeting survey, could encourage or require residents to complete the survey.

The positive feedback received from prepOSCE participants, both residents and evaluators, indicates that there is a need for a program like prepOSCE. This model has great potential for expansion and it is hoped that specialties outside of neurology will take it up.

Plans to improve the program next year include minor edits to the format to more closely match the Royal College OSCE (particularly if the College continues to use a virtual OSCE format); provision of evaluation forms in a digital format, which would be more streamlined for evaluators and would allow for immediate feedback to residents; the addition of in-depth practice sessions with evaluators beyond the roles review and technical check; and the development of more case scenarios to cover additional topics. A pre-meeting survey for evaluators could help to identify their comfort and experience creating and administering examinations, to ensure training resources are tailored to the needs of the evaluators leading into the training sessions.