Hiring and promotion in academic medicine depend on many factors, such as demonstrations of scholarly productivity, clinical expertise, seniority, teaching and administrative record, certifications and memberships, fellowship training, research funding, and institutional needReference Khan, Thompson and Taylor1. Among these factors, scholarly productivity is consistently regarded as a chief determinant for academic advancementReference Khan, Thompson and Taylor1–Reference Svider, Pashkova and Choudhry5.
Scholarly productivity can take many forms, including publications, grant capture, collaborations, presentations, patents, and reports. Perhaps the most reproducible and widely used means of assessing productivity is through the analysis of publications in peer-reviewed journals. Tools from the statistical field of bibliometrics have been used to quantify publication output, from simple metrics such as number of publications and citations, to more sophisticated metrics like h-index, r-index, m-quotient, and tier score of individual articlesReference Jamjoom, Wiggins, Loan, Emelifeoneu, Fouyas and Brennan6,Reference Lozano, Tam, Kulkarni and Lozano7 . Bibliometrics in neurosurgery is a growing area of study and has been used for the purposes of providing benchmarks, hiring and promotion, and ranking of individuals, institutions, residency programs, and fellowship programsReference Khan, Thompson and Taylor1,Reference Ajmera, Lee and Schultz8–Reference Ponce and Lozano10 .
Many neurosurgeons choose to undertake additional graduate degree training either before, during, or sometimes after residency. Previous work has shown that neurosurgeons with Masters and PhDs are more likely to find employment at an academic centerReference Choi, DeLong, DeLong, Friedman and Sampson11–Reference Tso, Max Findlay, Lownie, Chris Wallace, Toyota and Fleetwood13. Moreover, many recent academic job postings have required applicants to have a graduate degree as a mandatory condition of employment. What remains to be determined is whether an association exists between graduate degree training and future research productivity throughout an individual’s career. The purpose of this study was to perform an individual publication productivity analysis of academic neurosurgeons with an emphasis on graduate degree status.
Methods
All academic Canadian neurosurgical institutions were included in the study. An academic institution was defined as a center with a neurosurgery residency program registered with the Royal College of Physicians and Surgeons of Canada. The 14 institutions from west to east were: University of British Columbia, University of Calgary, University of Alberta, University of Saskatchewan, University of Manitoba, Western University, McMaster University, University of Toronto, University of Ottawa, McGill University, Université de Montréal, Université de Sherbrooke, Université Laval, and Dalhousie University.
A 5-year period was chosen from 2012 to 2016 to both account for yearly fluctuations in academic productivity as well as to allow papers a minimum of 2 years of time since publication to accrue citations, as of 2018. A list of active neurosurgeons was obtained both from program websites as well as emailing programs and individuals directly. The Royal College of Physicians and Surgeons of Canada public database was used to determine the year that each neurosurgeon obtained specialization status in neurosurgery and this was used as the first year of their career. Neurosurgeons who obtained Royal College status or departmental hiring in 2012 or later were excluded from the study, as well as those that retired in 2016 or earlier. Thus, all neurosurgeons included would have had the opportunity to publish in an academic environment throughout the entire 5-year period. Neurosurgeons were divided into three groups based on graduate degrees obtained: MD-only, MD-Masters, and MD-PhD. To minimize the number of comparisons and increase group size, all variants of a Masters (e.g. Masters of Public Health, Masters of Business Administration, Masters of Science) were included in the same group. In addition, the subject area or discipline of each individual degree was not considered. The timing of graduate degree training was dichotomized as either before the start of residency or after the start of residency. Timing was determined for each neurosurgeon either from department website profiles, public thesis repositories such as Theses Canada and university library databases, or by contacting individuals directly.
Publications were retrieved using Scopus, an abstract and citation database hosted by ElsevierReference Burnham14. The variables collected were the following: number of papers, CiteScore (an analog of impact factor), number of citations per paper, authorship position on each paper (either first/last or middle), whether the theme of the paper was clinical or basic science in focus, 5-year h-index, and 5-year r-index. The r-index is a modified h-index that weighs papers with authors in first/last position four times greater than middle authorship papersReference Lee, Xu and Dave9. The division between clinical and basic science articles was not always obvious, and in general, we categorized papers as being basic science if they involved in vitro or in vivo model systems or experimental developments in technology outside the realm of standard clinical practice. For articles co-authored by two or more neurosurgeons on our list, appropriate credit was assigned to each listed co-author according to authorship position.
Statistical analysis was completed using Prism (version 8, GraphPad). Normality testing was performed with the D’Agostino-Pearson normality test to determine the need for non-parametric testing, which was required for all subsequent analyzes. Between-group comparisons of data were carried out using the non-parametric Mann–Whitney U test or Kruskal–Wallis test with Dunn’s multiple comparisons test, as well as Fisher’s exact test for frequency distributions. Correlation was performed using Spearman’s test. Alpha was set to 0.05.
Results
In total, 2557 abstracts from 131 neurosurgeons were included in the analysis (Table 1). Of these neurosurgeons, 45.8% had no graduate degree (n = 60), 31.3% had a Masters (n = 41), and 22.9% had a PhD (n = 30). For the MD-Masters group, 19.5% (n = 8) completed their degree before residency, with 80.5% (n = 33) after residency started. For the MD-PhD group, 33.3% (n = 10) completed their degree before residency, with 66.7% (n = 20) after residency started.
Table 1: Descriptive statistics for all neurosurgeons included in the analysis. Central tendency and dispersion are median and interquartile range
The total number of papers were significantly different between groups (H = 25.5, df = 2, p < 0.0001, Figure 1). For all the following comparisons, values in parenthesis represent the median group value. The MD-Masters group published significantly more papers than the MD-only group (14.0 vs. 4.0, p = 0.002). The MD-PhD group published significantly more papers than the MD-only group (18.5 vs. 4.0, p < 0.0001). There were no differences between the MD-Masters and the MD-PhD group (p = 0.4). For clinical papers, there was a significant difference between groups (H = 21.5, df = 2, p < 0.0001). The MD-Masters group published significantly more papers than the MD-only group (13.0 vs. 4.0, p = 0.0008). The MD-PhD group published significantly more papers than the MD-only group (11.5 vs. 4.0, p = 0.0002). There were no differences between the MD-Masters and the MD-PhD group (p > 0.9). For basic science papers, there was a significant difference between groups (H = 29.6, df = 2, p < 0.0001). There were no differences between the MD-only and MD-Masters group (p = 0.4). The MD-PhD group published significantly more papers than both the MD-only (4.0 vs. 0.0, p < 0.0001) and the MD-Masters group (4.0 vs. 0.0, p = 0.0004). For number of papers with first/last authorship, there was a significant difference between groups (H = 27.5, df = 2, p < 0.0001). The MD-Masters group had significantly more papers than the MD-only group (3.0 vs. 1.0, p = 0.005). The MD-PhD group had significantly more papers than the MD-only group (8.5 vs. 1.0, p < 0.0001), but not the MD-Masters group (p = 0.1).
Figure 1: Comparative analysis of publications by 131 Canadian academic neurosurgeons by graduate degree status. Kruskal–Wallis with Dunn’s multiple comparison test. *p < 0.05, **p < 0.01, ***p < 0.001, ****p < 0.0001, n.s. not significant. MD+M: MD-Masters, MD+P: MD-PhD. Bars are median.
Given that individuals with few published papers overall can skew their publication metrics, for CiteScore, citations per paper, 5-year h-index and 5-year r-index, only those surgeons who published ≥1 paper/year were included in these analyses. These represented 48.3% of the MD-only group (n = 29), 75.6% of the MD-Masters group (n = 31), and 90.0% of the MD-PhD group (n = 27). For CiteScore, there was a significant difference between groups (H = 20.1, df = 2, p < 0.0001). The MD-Masters group did not differ from the MD-only group (p = 0.3). The MD-PhD group had a higher average CiteScore than both the MD-only (3.2 vs. 2.2, p < 0.0001) and the MD-Masters group (3.2 vs. 2.6, p = 0.01). For number of citations per paper, there was a significant difference between groups overall (H = 6.6, df = 2, p = 0.04). However, on post hoc testing, there were no significant differences between MD-only and MD-Masters (p > 0.9), MD-only and MD-PhD (p = 0.06), and MD-Masters compared to MD-PhD (p = 0.09). For 5-year h-index, there were significant differences between groups (H = 11.1, df = 2, p = 0.004). There were no differences between the MD-only and the MD-Masters group (p = 0.2). The MD-PhD group had a higher h-index than the MD-only (9.0 vs. 6.0, p = 0.003) but not the MD-Masters group (p = 0.4). For 5-year r-index, there were significant differences between groups (H = 17.1, df = 2, p = 0.0002). The MD-Masters group was not different than the MD-only group (p = 0.1). The MD-PhD group had a significantly higher r-index than the MD-only (8.4 vs. 3.3, p = 0.0001), but not the MD-Masters group (p = 0.09). Comparing the years active at the start of the analysis period, there was no significant differences between groups (H = 1.2, df = 2, p = 0.5).
At the end of our data collection, we could not ignore that a subset of individual neurosurgeons published numbers of papers that were orders of magnitude above the average for their individual group. We felt that the data should be re-analyzed after these outliers were removed, to generate a more accurate depiction of the association between graduate degrees and publication productivity in Canadian neurosurgeons. We chose to eliminate those that were >2 standard deviations from their respective group. A summary of the anonymized individuals that were removed are included in Table 2.
Table 2: Neurosurgeons removed as outliers from the dataset
With outliers removed, there were 115 neurosurgeons included in the modified analysis (Table 3), with 47.0% (n = 54) from the MD-only group, 30.4% (n = 35) from the MD-Masters group, and 22.6% (n = 26) from the MD-PhD group. For the MD-Masters group, 17.1% (n = 6) completed their degree before residency started, compared to 83.0% (n = 29) after residency started. For the MD-PhD group, 38.5% (n = 10) completed their degree before residency started, compared to 61.5% (n = 16) after residency started.
Table 3: Descriptive statistics for neurosurgeons included in the analysis with outliers removed. Central tendency and dispersion are median and interquartile range
In the modified analysis, there were significant differences between groups in terms of total number of papers (H = 26.9, df = 2, p < 0.0001, Figure 2). The MD-Masters group published significantly more papers than the MD-only group (10.0 vs. 3.5, p = 0.001). The MD-PhD group published significantly more papers than the MD-only group (15.5 vs. 3.5, p < 0.0001), but not the MD-Masters group (p = 0.4). For clinical papers, there were significant differences between groups (H = 22.7, df = 2, p < 0.0001). The MD-Masters group published significantly more clinical papers than the MD-only group (9.0 vs. 3.0, p = 0.0005). The MD-PhD group published significantly more clinical papers than the MD-only group (10.0 vs. 3.0, p = 0.0001), but not the MD-Masters group (p > 0.9). For basic science papers, there were significant differences between groups (H = 27.9, df = 2, p < 0.0001). There were no differences between the MD-Masters and MD-only groups (p = 0.9). The MD-PhD group published significantly more basic science papers than the MD-only group (3.0 vs. 0.0, p < 0.0001) and the MD-Masters group (3.0 vs. 0.0, p = 0.0002). For papers with first/last authorship, there were significant differences between groups (H = 30.2, df = 2, p < 0.0001). The MD-Masters group published significantly more papers than the MD-only group (3.0 vs. 1.0, p = 0.003). The MD-PhD group published significantly more papers than the MD-only group (6.5 vs. 1.0, p < 0.0001), but not the MD-Masters group (p = 0.1).
Figure 2: Comparative analysis of publications by 115 Canadian academic neurosurgeons by graduate degree status with the outlier neurosurgeons removed. Kruskal–Wallis with Dunn’s multiple comparison test. *p < 0.05, **p < 0.01, ***p < 0.001, ****p < 0.0001, n.s. not significant. MD+M: MD-Masters, MD+P: MD-PhD. Bars are median.
Including only those neurosurgeons with ≥1 paper/year, this accounted for 42.6% (n = 23) of the MD-only group, 71.4% (n = 25) of the MD-Masters group, and 88.5% (n = 23) of the MD-PhD group. Regarding CiteScore, there were significant differences between groups (H = 14.0, df = 2, p = 0.0009). The MD-only group did not differ from the MD-Masters group (p = 0.7). The MD-PhD group had significantly higher CiteScores than the MD-only group (3.2 vs. 2.1, p = 0.0008) and the MD-Masters group (3.2 vs. 2.6, p = 0.03). Regarding citations per paper, there were no significant differences between groups (H = 3.8, df = 2, p = 0.2). For 5-year h-index, there were significant differences between groups (H = 11.5, df = 2, p = 0.003). The MD-Masters group did not differ from the MD-only group (p = 0.2). The MD-PhD group had a significantly higher h-index than the MD-only group (8.0 vs. 5.0, p = 0.002), but not the MD-Masters group (p = 0.3). For 5-year r-index, there were significant differences between groups (H = 22.6, df = 2, p < 0.0001). The MD-Masters group had a significantly higher r-index than the MD-only group (5.1 vs. 2.7, p = 0.03). The MD-PhD group had a significantly higher r-index than the MD-only group (7.5 vs. 2.7, p < 0.0001), but not the MD-Masters group (p = 0.06). Regarding the years active since start of the analysis, there were no significant differences between groups (H = 2.1, df = 2, p = 0.3).
We next addressed whether the timing of graduate degree training was associated with any of the bibliometrics that were measured (Figure 3). Due to the overall low numbers of neurosurgeons who obtained graduate degrees before residency, we decided to combine the MD-Masters and MD-PhD groups for this portion of the analysis, using the modified dataset with outliers excluded. We also used the Fisher’s exact test to compare the proportion of MD-Masters to MD-PhD neurosurgeons in the before and after groups, and while there was a higher percentage of MD-PhD neurosurgeons in the before group, the proportions were not significantly different between the two groups (p = 0.08). Normality testing again failed for all the variables measured, and thus we used the non-parametric Mann–Whitney U test to compare the before and after groups. There were no differences between the two groups in terms of number of total papers (U = 313.0, p = 0.4), clinical papers (U = 298.5, p = 0.3), basic science papers (U = 344.5, p = 0.8), number of first/last author papers (U = 245.5, p = 0.07), CiteScore (U = 144.5, p = 0.09), citations per paper (U = 215.5, p > 0.9), 5-year h-index (U = 198.5, p = 0.7), 5-year r-index (U = 204.5, p = 0.8), or years active since start of the analysis (U = 309.0, p = 0.4).
Figure 3: Analysis of publications by 61 Canadian academic neurosurgeons with graduate degrees after removal of outliers and stratified by timing of graduate training. Mann–Whitney U test. n.s. not significant, Before: Graduate degree was completed before the start of residency, After: Graduate degree was completed after the start of residency. Bars are median.
Lastly, we asked if an association exists between length of an individual’s career and recent publication productivity (Figure 4). We performed a Spearman correlation analysis between the years active at the start of the analysis and the number of papers published in 5 years. There was no significant correlation between length of an individual’s career and their number of recent papers published (r = 0.02, p = 0.9).
Figure 4: Correlation analysis of 115 Canadian academic neurosurgeons between years active at start of analysis period and recent 5-year number of total publications. Spearman test. n.s. not significant.
Discussion
The goal of this study was to examine the role of graduate degree training on future publication productivity in academic neurosurgery. Our overall analysis with all neurosurgeons included suggested that neurosurgeons with MD-Masters degrees outperformed their MD-only colleagues in several quantitative metrics used: the number of total papers, clinical papers, and papers with first/last authorship. However, qualitative metrics such as CiteScore, citations per paper, h-index, and r-index were not significantly different. Neurosurgeons with MD-PhDs, by contrast, outperformed the MD-only group in almost all metrics, except citations per paper. Also, neurosurgeons with MD-PhDs outperformed their MD-Masters colleagues in terms of number of basic science papers and CiteScore.
It became clear during data collection and analysis, however, that we could not ignore the significant contributions from a select few neurosurgeons as potential outliers that could impact the entire dataset. Most of the outliers from the MD-Masters group, and all of the outliers from the MD-PhD group, were from the University of Toronto, a site that is among the top ranks of North American neurosurgical centers in terms of publication outputReference Lozano, Tam, Kulkarni and Lozano7,Reference Ponce and Lozano10 . We felt the data needed to be re-analyzed without the outliers included to provide a closer approximation of the average academic Canadian neurosurgeon output. With these outliers removed, the previously seen associations with graduate degree status were largely preserved. The only difference between the two analyses was that, with the outliers removed, the 5-year r-index was significantly higher in the MD-Masters group compared to the MD-only group. These refined data confirmed that for the vast majority of academic Canadian neurosurgeons, there still is a meaningful association between graduate degree status and publication productivity. We addressed the timing of graduate degree obtainment by comparing those with Masters or PhDs completed before starting residency to those after starting residency and found no significant differences for any of our metrics collected. We also found that there is no correlation between length of career and recent publication output for academic neurosurgeons in Canada.
The number of neurosurgical trainees pursuing PhDs is steadily increasing, from 10% to 26% over the past two decades at high ranking academic centers in the United StatesReference Choi, DeLong, DeLong, Friedman and Sampson11. A recent Canadian survey similarly found 25% of trainees obtaining PhDs and 33% obtaining a MastersReference Khan, Thompson and Taylor1,Reference Iorio-Morin, Ahmed and Bigder15 . There is no doubt that pursuing a graduate degree confers benefits useful to an academic neurosurgeon in general. In the United States, having a Masters or PhD has been associated with obtaining an academic position and advancement in academic rankReference Iorio-Morin, Ahmed and Bigder15. Interestingly, among those with a Masters, only a Masters of Science (and not MBA or MPH) has been associated with hiring to an academic practice in the United StatesReference Khalafallah, Jimenez and Tamargo12. Other studies have already reported the link between PhDs and increased publication output including h-index and m-quotientReference Wilkes, Akram, Hyam, Kitchen, Hariz and Zrinzo16. There are many other benefits that can be obtained from graduate training that are not necessarily included in a publication record, such as skills in teaching, collaboration, scientific exploration, knowledge translation, management, academic leadership, and grant capture. Grant capture in particular could also be a useful metric, given the large number of funding agencies that release competition results to the public, as well as the quantifiable number of grants obtained and dollar value allotments that could be used for comparison. We did not specifically look at grant capture in this study as we felt it was beyond the scope of the objectives. However, for the purposes of hiring committees, our data certainly suggest that graduate degree status of applicants could be used as a potential predictor of future publication productivity.
While a direct comparison between neurosurgeons from different countries was not the objective of the current study, a few parallels can be drawn from the existing literature. Ponce and LozanoReference Ponce and Lozano10 looked at departmental h-indexes for 99 academic institutions in Canada and the Unites States and found a positive correlation between department faculty size and departmental h-index. However, this correlation held true for both MD-only faculty and MD-PhD faculty, suggesting no increased output at the departmental level for having more staff with graduate degrees, which would disagree with our overall findings. Conversely, it may speak to the departmental environment and publishing culture as a key determinant for productivity, more than an individual’s graduate degree status. For example, we identified certain MD-only neurosurgeons with double the average number of publications, and, conversely, certain MD-PhD neurosurgeons with zero publications, over the 5-year period. Khan et al. performed a large publication analysis for 1225 individual academic neurosurgeons from the United States, but graduate degree status was not considered and thus cannot be comparedReference Khan, Thompson and Taylor1. Wilkes et al. performed an individual publication productivity analysis for neurosurgeons from the United Kingdom and found a positive correlation between h-index and the presence of an “MD” (equivalent to Masters, 16.8% of their neurosurgeons) or the presence of a PhD (11.7% of their sample). There are approximately double the percentage of Canadian neurosurgeons with Masters equivalent and PhD degrees, and while we found an effect for h-index for MD-PhD neurosurgeons, we did not for MD-Masters. Perhaps in a jurisdiction with lower saturation of graduate degrees, the magnitude of association between graduate degrees and publication productivity may be stronger. Wilkes et al. also looked at overall h-index, which is influenced by the length of an individual’s career, and those who undertook graduate training would have had more potential publishing years. Using incremental periods like 5-year h-index are therefore more meaningful for evaluating recently published work that is unbiased by the length of potential publishing years.
There are several limitations to the current study. According to the publicly available Canadian Medical Association Physician Workforce Survey, there were 333 neurosurgeons employed in Canada in 2018. Our study included only 121 neurosurgeons, about a third of the total active number. Reasons for this include limiting our analysis to only neurosurgeons employed at academic centers having a residency program. As well, we excluded anyone achieving specialization in neurosurgery or appointment to an academic position in 2012 or later, which, according to the publicly available Royal College of Physician and Surgeons of Canada database, would be approximately a third of practicing neurosurgeons in the country. We also did not account for those neurosurgeons with research experience outside a formal graduate training program (e.g., research fellowships), or neurosurgeons with degrees partially completed but not conferred. While we found no difference in the timing of graduate degree training on future productivity, we also did not quantify the number of publications published before or during residency, when those degrees were obtained. Daniels et al. have shown that only publications generated during residency, and not pre-residency, correlate with obtainment of a neurosurgical career at an academic siteReference Daniels, Garzon-Muvdi and Maxwell17. It would be interesting to see if those measures correlate with future productivity once an academic position has been obtained. Additionally, we did not address the impact of sub-specialization in our cohort, given prior data suggesting that neurosurgeons in the fields of cerebrovascular, neuro-oncology, and radiosurgery have the highest publication productivityReference Khan, Thompson and Taylor1.
As a final point, it is worth noting that mere co-authorship on a manuscript does not clearly indicate what role, if any, an individual had in the production of the manuscript. The number of authors on neurosurgical papers, and science in general, has steadily increased over timeReference Glynn, Kerin and Sweeney18. While in some instances this reflects larger projects from bigger teams, honorary or gift authorship is certainly still prevalentReference King19. Journals have attempted to mitigate this issue by providing transparent definitions of authorship requirements, but there is no way of knowing if these are being followed in every case. Outside of the first/last authorship designation, which likely indicates greater involvement and oversight in a published work, our study cannot account for the specific contributions of neurosurgeon authors to their published works.
Conclusion
The attainment of a graduate degree, regardless of timing in relation to residency training, has an important association with future publication productivity for academic neurosurgeons. These findings should benefit hiring committees for academic positions in neurosurgery in Canada and similar jurisdictions elsewhere.
Disclosures
The authors report no conflict of interest concerning the materials or methods used in this study or the findings specified in the paper.
Statement of authorship
MBK conceived the idea, generated and analyzed the data, and wrote the manuscript. CN and ARR collected data and contributed to the writing and editing of the manuscript. TS supervised the project and made substantial contributions to editing and writing of the manuscript.
