Neurosurgeons’ Perspectives on Vascular Entrustable Professional Activities

Abstract:

This article discusses subspecialty Canadian neurosurgeons’ perceptions of entrustable professional activities (EPAs) assessments and variabilities prior to the implementation of the Competence by Design (CBD) system in Canada. Vascular neurosurgeons were asked to reflect on how they would evaluate and give feedback to neurosurgery residents concerning the EPA “Performing surgery for patients with an intracranial aneurysm.” Interviews were transcribed and analyzed using a deductive approach. Themes were derived from these interviews and reflected on the subjectivity and biases present in the EPA assessment forms. Indeed, faculty may require more training in the transitioning to a CBD evaluation system.

Résumé :

Point de vue de neurochirurgiens sur les activités professionnelles confiables ciblant les lésions vasculaires.

Il sera question, dans l’article, du point de vue de neurochirurgiens au Canada sur les évaluations fondées sur des activités professionnelles confiables (APC) et de leur variabilité avant la mise en œuvre du système de Compétence par conception (CPC) au pays. Nous avons demandé à des neurochirurgiens vasculaires de réfléchir sur la manière dont ils évalueraient des résidents dans cette surspécialité, à l’aide de l’APC portant sur l’opération d’un anévrisme intracérébral, et sur la manière dont ils leur donneraient de la rétroaction. Les entretiens ont d’abord été transcrits, puis analysés selon une démarche déductive. Les thèmes retenus ont été tirés de ces entretiens et ont fait ressortir la subjectivité et les préjugés présents dans les formulaires d’évaluation des APC. En fait, il se peut que les enseignants aient besoin de formation supplémentaire durant le passage à un système d’évaluation par compétence par conception.

In July 2017, the Royal College of Physicians and Surgeons of Canada (RCPSC) implemented a Competence by Design (CBD) system of evaluation throughout all Canadian residency programs in order to mitigate knowledge gaps, enhance preparedness for independent practice, and ascertain when practice demands new abilities or skills. ¹ The CBD system is a hybrid competence-based medical education system and provided a new approach to the country’s postgraduate medical education. ² To achieve an intra-hospital context, milestones and entrustable professional activities (EPAs) were created by each specialty competence committee in the RCPSC followed by extensive faculty development efforts. Englander and colleagues (2017) defined EPAs as “essential task[s] of a discipline (profession, specialty, or subspecialty) that a learner can be trusted to perform without direct supervision.” ³ However, it is unclear how many subspecialists were included in the EPA development process, how surgical subspecialists would interpret EPAs in their discipline, and how those EPAs integrate into a CBD framework. To gain insight into this issue, this study examined the differences in the interpretation of a single EPA by subspecialty neurosurgery attendings and identified obstacles to the implementation of the CBD system from the perspective of these faculty.

We recruited vascular neurosurgeons from different neurosurgery residency programs in Canada. The recruitment period happened prior to the launch of CBD in the neurosurgery residency programs (January–March 2019). They were interviewed to capture their understanding of CBD. The interview questions were related to one of five EPAs specific to vascular neurosurgery from the Canadian neurosurgical residency programs CBD system. The vascular EPA was “Performing surgery for patients with an intracranial aneurysm.” Vascular neurosurgeons were asked to reflect on how they would evaluate and provide feedback to residents inquiring about an evaluation on this activity. Interview questions were designed to allow the interviewer to probe participant answers where possible. The interview guide was piloted on two different individuals (one general surgeon and one pediatric neurosurgeon) prior to starting the interviews to assure all the questions were clear and non-biased. Finally, individual interviews occurred remotely through videoconferencing (Zoom Video Communications, San Jose, CA) and were audio-recorded. Data were transcribed verbatim by one of the authors (MC).

Open coding was performed on one sample interview by two coders (MC and SKM), and the development of a codebook occurred as part of data reduction. The same two coders reviewed the codebook and agreed on its terms and on code definitions after various iterations. Two external reviewers (JRC and RP) verified the codebook to assure its precision and clarity. All interview transcripts were then independently coded at the sentence level by the two coders (MC and SKM) using QSR International’s NVivo 12 software. The coding was calibrated with the sample interview. Subsequent categories were created for thematic analysis utilizing inductive approach and performing axial coding with two other independent reviewers (NO and RP). The Conjoint Health Research Ethics Board (CHREB), University of Calgary, reviewed and approved the research protocol REB19-0453.

Vascular neurosurgeons described several elements influencing the level of trust they grant to residents in their program (Table 1). Most of them had a general idea of what an entrustable resident should be during residency training. Participants stated that the fund of knowledge and technical skills of a resident are highly important aspects for entrustability “Surgical anatomy is very important and that’s fundamental. The other aspects in terms of techniques, how are they handling their instruments? How are they retracting the brain?” (Surgeon 5) Similarly, vascular neurosurgeons judged the amount of independence to afford individual residents based on their previous surgical encounters with them and the necessity of the repetition of a task.

Table 1: Definition of themes derived from qualitative analysis

Entrustability is also influenced by concerns about patient safety and the likelihood of medicolegal involvement in the task to perform such as the clipping of an intracranial aneurysm when compared to a less complex task (i.e. a burrhole). One surgeon commented on the balance between patient safety and EPAs and noted “Independent to me would be the bar that I would set on how I feel they are able to do it on their own, […] but not meaning alone. It’s really about whether it’s translatable to independent practice.” (Surgeon 4) The level of residency training is another factor influencing patient safety. A senior resident as compared to a first-year resident should demonstrate more expertise and have gained more trust from the faculty.

Specifically related to the task of clipping an aneurysm, most interviewees responded that it requires multiple skills and that putting a clip on an aneurysm is simply the final step of a much more complex neurosurgical operation. However, they all had an opinion on how to evaluate a resident on this specific EPA regardless of the details provided to define the EPA. One participant noted “I will evaluate them as I evaluate them now. I don’t think the EPA per se changes the behavior of the evaluation of the procedure.” (Surgeon 2) Similarly, when the neurosurgeons explained what EPAs and milestones meant, either they did not know enough on the subject matter to describe it or provided a definition according to their own knowledge of the concepts. To overcome this issue, a consistent method of constructive feedback to the resident physicians may be helpful to ensure that both the faculty and resident physician have a shared mental model of the EPA and milestone.

This study explored subspecialty neurosurgeons understanding of a specific EPA as well as how they evaluate their trainees in the new CBD era of assessment. When discussing a single EPA, there are significant biases and variabilities when it comes to evaluating residents based on both trainee intrinsic and extrinsic factors. Most neurosurgeons would continue to grade residents the same way they have been doing prior to CBD, independently of EPAs’ assessment form’s information. Helping faculty understand questions such as “why are we changing?” or “how can we do this well?” may not only facilitate faculty knowledge about CBD training but also improve their approach with EPA assessments. ⁴ Interestingly, vascular neurosurgeons also expressed their concerns of CBD related to patient safety, which is in contrast with the main idea and rationale behind CBD to support quality and safety concerns in postgraduate medical practice. ⁵ These concerns might reflect a wrong interpretation of the “I did not need to be there” level of high entrustability included in the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) scale that is being used to grade residents in CBD. ⁶ Such discrepancies in interpretation of EPA grading should be addressed to perhaps develop simulated surgical scenarios specific to neurosurgery that fosters evaluators’ participation or close observation of the resident. These findings may or may not hold true once neurosurgeons are more familiar with the educational process.

The results of this article describe subspecialty neurosurgeons’ perceptions of EPA and competency-based assessments at a pivotal time in Canadian neurosurgery education when none of them had been involved in CBD before. These findings can translate to other specialties about to shift to competency-based medical education and lead to important lessons for faculty development and engagement and need for consistent constructive feedback mechanisms to resident physician learners.