Needs Assessment for Incoming PGY-1 Residents in Neurosurgical Residency

Abstract

Background: Residents must develop a diverse range of skills in order to practice neurosurgery safely and effectively. The purpose of this study was to identify the foundational skills required for neurosurgical trainees as they transition from medical school to residency. Methods: Based on the CanMEDS competency framework, a web-based survey was distributed to all Canadian academic neurosurgical centers, targeting incoming and current PGY-1 neurosurgical residents as well as program directors. Using Likert scale and free-text responses, respondents rated the importance of various cognitive (e.g. management of raised intracranial pressure), technical (e.g. performing a lumbar puncture) and behavioral skills (e.g. obtaining informed consent) required for a PGY-1 neurosurgical resident. Results: Of 52 individuals contacted, 38 responses were received. Of these, 10 were from program directors (71%), 11 from current PGY-1 residents (58%) and 17 from incoming PGY-1 residents (89%). Respondents emphasized operative skills such as proper sterile technique and patient positioning; clinical skills such as lesion localization and interpreting neuro-imaging; management skills for common scenarios such as raised intracranial pressure and status epilepticus; and technical skills such as lumbar puncture and external ventricular drain placement. Free text answers were concordant with the Likert scale results. Discussion: We surveyed Canadian neurosurgical program directors and PGY-1 residents to identify areas perceived as foundational to neurosurgical residency education and training. This information is valuable for evaluating the appropriateness of a training program’s goals and objectives, as well as for generating a national educational curriculum for incoming PGY-1 residents.

Résumé

Évaluation des besoins des résidents qui commencent le programme de formation en neurochirurgie.Contexte: Les résidents doivent développer différentes compétences afin de pratiquer la neurochirurgie de façon sûre et efficace. Le but de cette étude était d’identifier les compétences fondamentales dont les résidents en neurochirurgie ont besoin au moment où ils passent de l’école de médecine à la résidence. Méthode: Une enquête en ligne basée sur le cadre de compétences CanMeds a été distribuée à tous les centres académiques de neurochirurgie. Cette enquête ciblait les nouveaux résidents et ceux de première année ainsi que les directeurs de programme. Les répondants devaient évaluer l’importance de différentes compétences au moyen de l’échelle de Likert, soit des compétences cognitives (par exemple, la prise en charge de l’élévation de la pression intracrânienne), techniques (par exemple, effectuer une ponction lombaire) et comportementales (par exemple, obtenir un consentement éclaire) exigées des résidents de première année. Résultats: Parmi les 52 individus contactés, 38 ont répondu à l’enquête. Parmi eux, 10 étaient des directeurs de programme (71%), 11 étaient des résidents de première année (58%) et 17 étaient de nouveaux résidents (89%). Les répondants ont mis l’emphase sur des compétences opératoires comme le respect d’une stricte asepsie et d’un positionnement adéquat du patient ; des compétences cliniques telles la localisation d’une lésion et l’interprétation d’examens de neuro-imagerie ; des compétences concernant la prise en charge de scénarios fréquents comme l’augmentation de la pression intracrânienne et le status epilepticus et des compétences techniques comme la ponction lombaire et la mise en place d’un drain ventriculaire externe. Les réponses en texte libre concordaient avec les résultats obtenus à l’échelle de Likert. Discussion: Nous avons effectué une enquête auprès des directeurs de programme de neurochirurgie Canadiens et des résidents qui entrent en première année de résidence pour identifier les sujets perçus comme étant fondamentaux à l’enseignement et à la formation en neurochirurgie. Cette information est précieuse pour évaluer si les buts et les objectifs d’un programme d’entrainement sont appropriés ainsi que pour développer un curriculum éducatif national pour les résidents qui commencent l’entrainement en neurochirurgie.

Neurosurgeons develop a myriad of cognitive, technical and behavioural skills during residency training, which they are subsequently required to use in high-pressure situations during clinical practice. In recent years, neurosurgical residency training has been restructured to limit residents’ duty hours; combined with an increased emphasis on operating room efficiency and patient safety, these changes have limited the time available for direct surgical instruction. ^{1 - 3} Given the heterogeneity in undergraduate medical school experiences at the individual and institutional levels, ^{4 - 6} many fundamental skills required of neurosurgical residents may not be taught in medical school. This may exacerbate the already steep learning curve that trainees face at the start of residency. ^{5 , 7}

In both the United States of America (US) and Canada, national accreditation bodies have called for the development of residency curricula that incorporate competency-based goals and objectives at each educational level, ⁸ based on established frameworks – namely the Accreditation Council for Graduate Medical Education (AGCME) Competencies in the US and the CanMEDS competencies in Canada. However, an ongoing challenge is to specify what competencies should be achieved at each level of training. Published data detailing specific objectives for training at the incoming resident level – especially in neurosurgery – are lacking.

In light of this gap in the literature, we developed a national needs assessment survey based on the CanMEDS competency framework. By surveying a national base of incoming neurosurgical residents, current PGY-1 neurosurgical residents and residency program directors, we sought to identify the foundational skills required of trainees entering neurosurgical residency. The resulting set of cognitive, behavioral and technical skills may be used not only to define competency milestones for incoming and junior neurosurgical residents, but also to develop training programs to address knowledge gaps as junior residents begin neurosurgical training.

Methods

Identifying Survey Respondents

In selecting survey participants, we sought to include those who would have a vested interest in optimizing the incoming PGY-1 resident experience. Thus, all 14 Canadian neurosurgical program directors were invited to participate, to capture the perspective of educators with an informed opinion on the overarching training needs of PGY-1 residents. We invited all 19 incoming PGY-1 residents, since they would be directly impacted by potential educational reforms resulting from the survey and could reflect on gaps in their medical training related to neurosurgical knowledge and skills. Finally, we invited all 19 Canadian residents currently completing their PGY-1 neurosurgical training year, to provide perspective on the fundamental skills required at the onset of neurosurgical training.

After obtaining ethics approval from the Capital Health District Authority Research Ethics Board in Halifax, NS, a 69 item web-based questionnaire was developed using the SelectSurvey software (Atomic Design, Overland Park, KS). The survey was sent to each neurosurgical program director and neurosurgical training program administrator in Canada, who were asked to distribute the letter of invitation and survey to the incoming and current PGY-1 residents admitted to their respective programs. Informed consent was implied from participants who completed the questionnaire.

Respondents were sent a reminder email every two weeks over a four-week period before the survey was closed. Survey responses were stored online in a secure database.

Survey Design

The content of the survey was developed through consensus among the authors: a practicing Canadian neurosurgeon (DBC), a medical educator (MCM), and a Canadian junior and senior neurosurgical resident (DMB and FH, respectively).

The survey was divided into 61 Likert-response and 8 open-ended response questions framed around the CanMEDS competencies. ¹ Likert response questions asked respondents to rank the importance of various cognitive, behavioural and technical skills on an achored 5-point scale: “very important” (1), “important” (2), “neutral” (3), “unimportant”(4), and “very unimportant” (5). These 61 questions were further categorized into seven subsections based on different domains: (1) operating room knowledge and skills; (2) basic anatomical and clinical principles; (3) basic anatomical, physiological and pathological principles; (4) management principles; (5) technical operative skills; (6) behavioural and communication skills; and (7) inter-professional communication skills. The eight open-ended questions invited participants to provide additional skills in each of the seven subsections and again at the conclusion of the survey, using free-text response boxes.

Data Analysis

The analysis of Likert scale data remains controversial in the academic medical literature. Likert survey questions technically collect ordinal data, as a set interval cannot be assumed between values such as “Very Important” and “Important”. ⁹ However, many authors have highlighted the value of interpreting Likert responses as interval data ¹⁰ and, in fact, this is common practice within the Neurosurgical literature. ^{2 , 11 , 12} In light of this controversy, for each of the 61 Likert-response items we opted to report both the mean response, as well as the frequency of the response “very important” among all survey participants and within each response group (incoming residents, current residents and program directors). Data collected from free-text responses to open ended questions were analyzed using descriptive methods and summarized into categories based on the main idea in each response. ¹³

To give the reader a sense of the relative importance of each item, we have organized the quantitative items in terms of their relative rankings within each subsection of the survey. Given the Likert data are ordinal in nature, we opted to analyze the data using non-parametric methods using the Kruskal-Wallis test. ⁹ For items where a significant difference was noted, post-hoc pairwise comparisons were carried out between the three groups using the Mann-Whitney test, using the Bonferroni method to correct for multiple comparisons.

Results

A total of 38 responses were collected out of a total of 52 possible respondents for an overall response rate of 73%. These included 10 program directors (10/14 or 71%), 11 Current PGY-1 residents (11/19 or 58%) and 17 incoming PGY-1 residents (17/19 or 89%). The responses from Likert-scale questions are detailed in Tables 1-7 and summarized by each of the seven survey categories below, whereas the data from open-ended questions are provided in Table 8 and summarized in the “Responses to Open-ended Questions” section below.

Table 1 Results from quantitative portion of survey describing operating room knowledge and skills

Includes means of Likert scale responses as well as frequency of “Very Important” responses. Data are presented in descending order according to the mean values for all responses. All=All responses from survey; PD=Program Directors; Inc=Incoming Residents; Cur=Current residents

Table 2 Results from quantitative portion of survey describing basic anatomical and clinical principles

^a A statistically significant difference between incoming residents and program directors was observed for this item (p<0.011, Mann-Whitney U Test). Includes means of Likert scale responses as well as frequency of “Very Important” responses. Data are presented in descending order according to the mean values for all responses. All=All responses from survey; PD=Program Directors; Inc=Incoming Residents; Cur=Current residents; CNS=central nervous system; XR=X-Ray; CT=computed tomography; MRI=magnetic resonsance imaging; CBC=complete blood count; CSF=cerebrospinal fluid

Table 3 Results from quantitative portion of survey describing Basic anatomical, physiological and pathological principles

Includes means of Likert scale responses as well as frequency of “Very Important” responses. Data are presented in descending order according to the mean values for all responses. All=All responses from survey; PD=Program Directors; Inc=Incoming Residents; Cur=Current residents

Table 4 Results from quantitative portion of survey describing principles of management

Includes means of Likert scale responses as well as frequency of “Very Important” responses. Data are presented in descending order according to the mean values for all responses. All=All responses from survey; PD=Program Directors; Inc=Incoming Residents; Cur=Current residents; CNS=central nervous sytem; VP=ventriculoperitoneal

Table 5 Results from quantitative portion of survey describing technical operative skills

^b A statistically significant difference between current residents and project directors was observed for this item (p<0.007, Mann-Whitney U Test). Includes means of Likert scale responses as well as frequency of “Very Important” responses. Data are presented in descending order according to the mean values for all responses. All=All responses from survey; PD=Program Directors; Inc=Incoming Residents; Cur=Current residents; VP=ventriculoperitoneal.

Table 6 Results from quantitative portion of survey describing behavioural and communication skills

Includes means of Likert scale responses as well as frequency of “Very Important” responses. Data are presented in descending order according to the mean values for all responses. All=All responses from survey; PD=Program Directors; Inc=Incoming Residents; Cur=Current residents; ER=emercgency response; ICU=intensive care unit; OR=operating room

Table 7 Results from quantitative portion of survey describing inter-professional communication skills

Includes means of Likert scale responses as well as frequency of “Very Important” responses. Data are presented in descending order according to the mean values for all responses. All=All responses from survey; PD=Program Directors; Inc=Incoming Residents; Cur=Current residents; CT=computed tomography

Table 8 Qualitative responses

Operating Room Knowledge and Skills

For cognitive and technical skills in the operating room, there was a consistent preference towards basic rather than specialized skills (Table 1). For instance, skills such as using an intra-operative microscope or neuro-navigational tools were found to be “very important” by only 18% and 39% of all survey respondents, respectively. In contrast, 66% and 63% of respondents cited sterile draping and patient positioning to be very important, respectively. In particular, 80% and 70% of program directors felt that emphasizing sterile draping and using a Mayfield headrest were very important skills. For all items in this subcategory, no significant differences in the responses between incoming resident, current residents and program directors were noted, suggesting general agreement on the relative importance of these items.

Basic Anatomical and Clinical Principles

As seen in Table 2, when asked about basic anatomical and clinical principles that should be emphasized for incoming PGY-1 residents, 82% of respondents found taking a neurosurgical history and performing an examination to be very important. Also highly rated were the interpretation of radiographic imaging (74%), establishing effective differentials (63%) and localizing lesions (55%). Reviewing anatomy of the spine, cortical surfaces, the ventricular system, cranial nerves and the skull base were less highly ranked. Of note, only 10% of program directors felt that reviewing peripheral nerve, skull base and cranial nerve anatomy was very important. In comparing between the three groups, a statistically significant difference was found for the item “Review of Bones in the Skull Base” (χ²=7.502; p<0.023). A post-hoc analysis for multiple comparisons found a significant difference between incoming residents and program directors (p<0.011), but not between current residents and program directors or between the two resident groups.

Basic Anatomical, Physiological and Pathological Principles

As seen in Table 3, respondents ranked reviewing hydrocephalus (53%) as the most important basic anatomical, physiological and pathological principle to review, followed closely by a review of hyponatremia and spinal pathology. Only 3% of respondents felt that reviewing congenital neuroaxis malformations was very important and 5% felt that reviewing functional neurosurgery was very important. Of note, none of the residents felt that a review of congenital malformations was very important. No significant differences between residents and program directors were observed.

Principles of Management

As seen in Table 4, all of the program directors felt that the management of a seizing patient was very important, while 91% of incoming and 60% of current PGY-1 residents found this to be very important. Many of the program directors (80%) felt that reviewing the management of raised intracranial pressure was very important. In contrast, only 32% of respondents felt that a review of pituitary apoplexy was very important. No statistically significant differences were observed.

Technical 0perative Skills

The highest ranked technical operative skills (Table 5) were performing a lumbar puncture (74%) and establishing hemostasis using operative instruments (74%). Of note, none of the program directors felt that using an endoscope to navigate surgical anatomy or reviewing the skills of blunt dissection was very important. In contrast, 45% of incoming residents and 41% of current residents felt that reviewing blunt dissection was very important; this difference was statistically significant (χ²=7.612; p<0.022). Post-hoc comparisons demonstrated a significant difference between current residents and program directors on this item (p<0.007).

Behavioral and Communication Skills

While none of the responses reached statistical significance, a general trend was seen where program directors seemed to rank communication skills higher than residents (Table 6). Of note, imparting bad news to families was ranked highest by program directors (80%), whereas obtaining informed consent was ranked highest by both incoming and current PGY-1 residents (64% and 71% respectively).

Inter-Professional Communication skills

As seen in Table 7, many program directors (70%) felt that communicating red-flags for post-operative craniotomy and spinal surgeries were very important. Residents also ranked these two communication skills highest, with 50% of incoming and 59% of current PGY-1 residents ranking them as very important. By contrast, only 10% and 24% of residents felt that emphasizing communicating referrals for lower back pain (i.e. discussing with referring physicians appropriate referrals for lower back pain) was important. Again, no significant differences were observed between groups.

Responses to Open-ended Questions

Free text responses to open-ended questions were categorized based on the knowledge or skill-type addressed in the comments. These categorized responses are provided in Table 8. Survey respondents provided a variety of suggestions for additional knowledge and skills that are relevant to PGY-1 training beyond those listed in the survey questions, although virtually no responses were repeated between survey participants. Of particular note were suggestions to incorporate non-technical basic operative skills like writing post-operative orders, reviewing appropriate behavior and following the World Health Organisation surgical safety checklist. Other skills include recognizing unstable patients, communicating urgent compared with non-urgent cases and appropriate charting techniques.

Discussion

Interpretation of Findings

We have collected data from a representative group of residents and program directors from multiple Canadian Neurosurgical training programs. Survey respondents have a vested interest in the education of junior neurosurgical residents and possess diverse and complementary perspectives. We achieved a good response rate, with 71% program directors and 73% of invited residents responding to the survey. Although the survey was designed and administered within a Canadian context, given the similarities between the competency frameworks employed by many national accreditations bodies (including the Canadian CanMEDS and the American ACGME competencies), the results are potentially applicable for neurosurgical training in other parts of the world.

It is notable that in the 61 Likert-response questions only two demonstrated significant differences between program director and resident responses. Post-hoc comparisons revealed that significant difference arose in ratings of the importance of “Blunt Dissection” between program directors and current residents, and the importance of skull base anatomy between program directors and incoming residents. The fact that response trends remained fairly consistent between the surveyed groups suggests perceptions of junior residents and program directors regarding incoming PGY-1 training needs were very similar.

Three general clusters of responses were found in the free text answers. First, some respondents identified specific knowledge deficiencies not otherwise listed in survey questions; second, respondents were interested in basic approaches to safe behaviour in the operating room, including professionalism and communication; and third, many respondents emphasized foundational communication skills, including understanding how to communicate during urgent and non-urgent scenarios.

A trend observed between survey categories and qualitative responses was respondents’ emphasis on the need for training basic rather than advanced skills. These results are not surprising, as such basic skills appropriately reflect the role a junior resident would play as part of the neurosurgical service and highlight many general competencies that can serve as building blocks for the development of more advanced cognitive, behavioural and technical skills later in residency.

Implications for Designing Neurosurgical Curricula

The Royal College of Physicians and Surgeons of Canada Specialty Training Committee in Neurosurgery has published a set of core requirements required of neurosurgical graduates at the completion of residency training. ¹⁴ However, the timing of various sub-skills and milestones required to achieve these objectives have not yet been specified. Thus, the approach to learning these skills and meeting expectations is currently left to the discretion of the individual learner, under the supervision of program directors at each training program. The results of this survey may provide a preliminary ‘blueprint’ for curricular milestones for incoming and junior neurosurgical residents, so they can focus their learning on a more limited scope of skills with the intention of developing a solid foundation to guide them through the later stages of residency training.

A national Neurosurgical training program utilizing simulation-based methods currently exists in the US, developed by The Society of Neurological Surgeons (SNS). The “Boot Camp” is designed to provide PGY-1 residents with a fundamental set of skills relevant for neurosurgical training. ¹⁵ Building on the experiences of the SNS course, the results of this survey have been used to guide curriculum development for a similar neurosurgical Rookie Camp for incoming PGY-1 residents in Canada, based on the CanMEDS framework (see Haji et al. in this issue). It is our hope that using such a needs-based approach will ensure the educational value of the program that is ultimately delivered.

Limitations

There are some important limitations to the current study. First, the survey questions were developed through informal consensus among all four authors, whose backgrounds included a practicing Canadian neurosurgeon, a medical educator, and two neurosurgical residents. However, the questions were not generated using a specific consensus-building methodology (e.g. modified Delphi method). In addition, the survey was not subjected to formal piloting with the target population, creating the potential for misinterpretation of survey items by participants. Notwithstanding these limitations, we maintain the questions do empirically represent a substantial sample of the cognitive, behavioral and technical skill set required for junior neurosurgical residents. Further, the substantial overlap between our questions and the subject matter covered at the recent ACGME Neurosurgery Boot Camp highlights skill-sets valued by both educators and learners. ^{12 , 15}

Second, the finding of only two items with statistically significant differences between residents and program directors may represent a flaw in the analytical tools used for our data set (see Data Analysis section), which may not be sensitive to detect true differences between resident and program director opinions. Alternatively, this finding may represent concordance of opinion between these two groups.

Third, we did not include intermediate or senior level residents as survey participants, reasoning that junior residents would be in the best position for identifying educational requirements for their peers. Senior residents may have provided additional insights; however, we maintain that the similarities between program directors’ and junior residents’ responses highlights that relevant educational needs for this cohort were likely captured with the existing participant pool.

Conclusion

Given today’s climate of increasing work-hour restrictions and decreasing intra-operative educational experiences, residency training is in the process of undergoing a substantial paradigm shift towards competency-based models of training. Concurrently, incoming residents struggle with heterogeneous medical training and a substantial learning curve that must be overcome to meet the expectations of junior residency. Our national survey identifies important technical skills to be mastered early in neurosurgical residency training. Our survey also suggests that junior residents and program directors agree that addressing specific cognitive and behavioral skills is important and should specifically focus on basic management principles both inside and outside of the operating room. Our findings may be helpful for individual training programs and also may serve as a basis for curriculum development of a national training initiative at the onset of neurosurgical residency training (see Haji et al. in this issue).