The Rockefeller Team Science Leadership training program: Curriculum, standardized assessment of competencies, and impact of returning assessments

Abstract

The ability to effectively lead an interdisciplinary translational team is a crucial component of team science success. Most KL2 Clinical Scholars have been members of scientific teams, but few have been team science leaders. There is a dearth of literature and outcome measures of effective Team Science Leadership in clinical and translational research. We focused our curriculum to emphasize Team Science Leadership, developed a list of Team Science Leadership competencies for translational investigators using a modified Delphi method, and incorporated the competencies into a quantitative evaluation survey. The survey is completed on entry and annually thereafter by the Scholar; the Scholar’s primary mentor and senior staff who educate and interact with the Scholar rate the Scholar at the end of each year. The program leaders and mentor review the results with each Scholar. The survey scales had high internal consistency and good factor structure. Overall ratings by mentors and senior staff were generally high, but ratings by Scholars tended to be lower, offering opportunities for discussion and career planning. Scholars rated the process favorably. A Team Science Leadership curriculum and periodic survey of attained competencies can inform individual career development and guide team science curriculum development.

Introduction

The overall vision of the Rockefeller University Center for Clinical and Translational Science (CCTS), supported by the National Institutes of Health Clinical and Translational Science Award (CTSA) program, is to develop, demonstrate, evaluate, and disseminate innovative programs to achieve translational success and to integrate these into a seamless “Learning Clinical Research Enterprise” that uses outcome data to drive quality improvement for the benefit of human health. The Rockefeller University 3-year Masters’ degree KL2 Clinical Scholars Program, which began in 2006 and is supported by both CTSA and University resources, is a key component of the CCTS, and its goal is to prepare translational team science leaders who are able to use their skills to improve human health [1]. It is designed to provide an optimal environment for junior translational investigators to develop team science and leadership skills by designing, conducting, analyzing, and reporting a human participant protocol under the supervision of an expert senior scientific mentor and a multidisciplinary group of Team Science Educators/Evaluators (TSE/Es) who comprise the Senior Staff of the CCTS. We have approximately 15–20 Scholars in the program at any time.

Our motivation to focus on Team Science Leadership is rooted in our belief that the ability to effectively lead an interdisciplinary translational team is a crucial component of team science success, supported by feedback from prior graduates who expressed insecurity about their Team Science Leadership skills immediately after graduation, and by our observation that despite our Scholars having extensive experience participating as team science members, they have little experience as team science leaders. While there is universal recognition of the importance of team science and team leadership in the conduct of clinical and translational science (CTS) [1–3], and an enormous literature from industry on team leadership and organization [2,4], there is a remarkable dearth of outcome measures of effective team leadership in CTS. Moreover, while the business literature on corporate leadership is robust, in our literature review, we could not identify any team leadership competencies that are specific for serving as a team leader of clinical and translational research. Specifically, PubMed searches in July 2021 for papers with the terms “team science outcome” or “team science competencies” in their titles yielded no papers. The 2011 CTSA Core Competencies in Clinical and Translational Research touched on Translational Teamwork and Leadership [5] but were very general and did not provide a mechanism to assess outcomes. Even a search of the NCI Team Science Toolkit [6] did not identify tools for outcome assessment of Team Science Leadership. To address these gaps, we enhanced our curriculum to emphasize Team Science Leadership, developed a list of team science leadership competencies specific for translational investigator team leaders, and implemented a Team Science Leadership quantitative evaluation survey to be completed annually by the Scholar, the Scholar’s primary mentor, and the TSE/Es. We also implemented a process to review the results with the Scholars to ensure that they receive feedback on their progress at a time when they can plan a course of action to ensure that they master the competencies before graduating. Analyzing the results of the survey has also provided crucial additional information that has enhanced our ability to: 1) teach team leadership skills, 2) provide opportunities for trainees to develop their skills via experiential learning, 3) evaluate trainees’ competencies, 4) provide feedback to trainees about their performance, 5) assess whether providing feedback results in improved performance, and 6) identify opportunities to strengthen individual components of team science training. This paper describes our Team Science Leadership curriculum, presents the process we followed to create the Rockefeller Team Science Leadership competencies and survey, and provides early results on psychometrics, means and variability of items, and lessons learned on the utility of the survey.

Methods

Team Science Leadership Curriculum

The Team Science Leadership curriculum is outlined in Table 1 and discussed in more detail in Supplementary Materials.

Table 1. Elements of the Team Science Leadership curriculum

We introduced the Team Science Leadership initiative to the Clinical Scholars by providing them with a Team Science Leadership Self-Assessment document with a series of questions about

Team Science Leadership for self-assessment (Table 2), and if they chose, discussion with their primary mentor and/or the leaders of the Clinical Scholars program. The document covers the Scholar’s experiences and/or views of current and past experience in participating in teams, developing and articulating a vision for a team, delegation of tasks, oversight of team members, decision-making, conflict resolution, models of leadership and role modeling, and self-reflection on the Scholars’ current strengths and weaknesses as a team leader. In addition, Scholars rate themselves on their Team Science Leadership Competencies (see below) on entry into the program and yearly using the same survey used by the TSE/Es. To broaden the Scholars’ scope in contemplating their roles as leaders, we searched for other leadership models that might be instructive for comparison. We selected a program on leadership, The Art of Teams: Achieving Excellence as Equals, that uses the relationship among the members of a string quartet as a paradigm of shared and shifting team leadership, reflecting the relative roles of the instruments as dictated by the music and the overall team structure of the quartet [7].

Table 2. Rockefeller University Clinical Scholars Team Science Leadership self-assessment survey

Team Science Leadership Competencies and Teams Science Leadership Survey

As TSE/Es, we collectively recognized the need to define the competencies for effective Team Science Leadership for translational research studies. By combining our domain-specific expertise and experience, we created a list of competencies that we assessed to be essential for an outstanding translational scientist to lead a multidisciplinary team in conducting a human participant research protocol (Supplemental Figure 1). All 15 Senior Staff members participated in a modified Delphi method [1,8,9] for generating and reviewing both the major domains and the items within domains. Over the course of 6 months of weekly meetings, the domains were first agreed upon, and then the items within domain were created, deleted, and revised until neither new items nor major revisions were suggested. These draft items were then shared with all of the current Scholars for feedback and revision. They were also shared with colleagues from Yale University and the University of Pennsylvania in a series of face-to-face meetings at each site. The list of competencies was finalized to begin the project, but with agreement that the competencies would need to be reviewed and modified periodically.

The final list converged on five domains: 1. Foundational Leadership Competencies containing nine items centered on creating and leading teams, creating and communicating a vision, building trust among team members, mentoring team members, and establishing lines of open communication; 2. Professionalism Competencies containing five items that describe aspects of responsibility and modeling ethical and supportive behavior; 3. Team Building and Team Sustainability containing 12 items that describe more granular level aspects of team communications, recognition of team member achievements, conflict resolution, seeking advice, and taking responsibility; 4. Appropriate Use of Resources and Execution of Study containing six items that describe elements of appropriate budgeting, staffing, overcoming obstacles, organizing continual quality improvement, and delegating responsibility appropriately; 5. Regulatory Accountability containing five items describing issues relating to compliance with Good Clinical Practice (GCP), federal and state laws and regulations, and issues related to data management, security, and sharing.

Prior to the Team Science Leadership initiative, we periodically reviewed our perceptions of the progress of individual Scholars during our weekly TSE/Es meetings. During these discussions, we appreciated the value of sharing our perceptions since we each observed the Scholars in different roles, interacting with different people, and performing different tasks. We thus recognized that by summing our individual observations we had a much more comprehensive assessment of the Scholars’ progress. We therefore set a goal of systemizing the information from all TSE/Es and combining it with the Scholars’ mentors’ evaluations and integrating it into a comprehensive plan for competency evaluation, feedback, and improvement. To achieve this goal, we developed a research protocol describing the evaluation of Scholars’ Team Science Leadership competencies and received approval from the Rockefeller Institutional Review Board prior to implementing the study. To evaluate the Team Science Leadership competencies of the Scholars, we converted the competencies into a survey in which each competency is rated based on a six-point Likert scale score (0–5), with additional space for free text comments (Table 3). We then implemented the survey in a REDCap [10] format to facilitate data collection and preserve the confidentiality of the reviews. Each Scholar evaluates herself or himself, each mentor evaluates her or his Scholar, and each TSE/E evaluates each Scholar across as many items and dimensions as are appropriate, based on content area and exposure to the Scholar’s activities in that area (Supplemental Figure 2) The evaluations by the TSE/Es, mentor, and Scholars are combined into a written report that includes 1. the aggregated evaluations by the mentor and TSE/Es, presented as the mean ± SD score for each competency, 2. an adjacent column with the Scholars’ self-evaluation scores, 3. an adjacent column listing the difference in the aggregated score and the Scholar’s score for each competency, and 4. free text, anonymized comments by the TSE/Es and mentor. After field testing the survey with one Scholar, we made minor revisions to the survey and to the REDCap administration platform.

Table 3. Team science competencies

Note: Scholar ratings less than or equal to 3.0 and TSE/E versus Scholar ratings greater than 1.0 are in bold.

The Return of Results process consists of the CTSA PI and the Director of the Clinical Scholars Program prereviewing the report to make sure that any negative comments are framed constructively. They then meet with the Scholar and the Scholars’ mentor to review the findings and focus on identifying areas where the Scholar thinks she or he needs additional training or opportunities to develop the competencies. The review also focuses on comparing the Scholars’ self-evaluation on each competency to the aggregate evaluation by the Scholar’s primary mentor and all of the TSE/Es. Where appropriate, the group identifies additional resources or programs that may help the Scholar develop the competency. After the meeting, Scholars are asked to complete a short anonymous survey (Table 4) providing feedback on the value of participating in the survey and the Return of Results process.

Table 4. Survey for clinical scholars to report the value of receiving results from the team science competency survey

The first administration of the competency survey was in the Fall of 2019, followed by a second administration in the Fall of 2020, which included the incoming scholars of the 2020 cohort.

Statistical Analysis

Means and standard deviations were calculated for each item using the 0–5 scale, which spanned from “Not At All Well” to “Extremely Well,” respectively, so that a higher score indicated a greater sense of competence of the item. If the TSE/E or mentor evaluators did not have a basis for making a judgment, they selected “Not Applicable” and the result was set to missing. Equal weighted scales for each domain were calculated if greater than 50% of the items in the scale were non-missing. Data from Scholars and TSE/Es were then merged to calculate item differences as the mean of the combined scores of the TSE/Es and mentor minus the Scholar score so that a positive difference indicated that the TSE/E and mentor scores were higher than the Scholar’s score.

A Cronbach’s alpha was computed for each subscale to assess the internal consistency and inform reliability. To help better understand the scale structure, a factor analysis with principal component dimension extraction followed by orthogonal rotation was employed within each domain. Subsequent scree plots of the eigenvalues were examined to assess the dimensionality of the subscales. All analyses were conducted in SAS Studio V94.

Results

A total of 31 Team Science Raters (which included TSE/Es and mentors) made a total of 115 ratings across 15 scholars (11 unique, 4 rated twice). The number of TE/Es and mentors rating of the Scholars ranged from 2 to 11. All of the mentors evaluated their Scholars. The mentors’ ratings were incorporated into the aggregated TSE/Es ratings to preserve anonymity. The average Cronbach’s alpha coefficient was 0.85 (range 0.71–0.97), where cells with fewer items had predictably lower coefficients. Visual assessment of the scree plots from the exploratory factor analysis supports a single-factor solution within each domain, where the first eigenvalue of each scale accounted for greater than 50% of the variance and the second typically less than 15%.

Table 3A presents the nine items that comprise the Foundational Leadership Competency scale, along with the average ratings by TSE/Es–mentors and Scholars, and the average difference between the ratings. In general, the TSE/Es mentors scored the Scholars higher that the Scholars scored themselves; TSE/Es mentors thought Scholars excelled at “Developing and nurturing collaborations and external collaborations” and had the most room to improve in “Anticipating obstacles and devising strategies to overcome them”, while Scholars in general thought they excelled at “Creating a culture that values and supports diversity”, and in “Developing and nurturing collaborations and external relationships”, while they felt least competent in “Exploring opportunities for growth and development”. The largest disparity in ratings between TSE/Es–mentors and Scholars in this scale was in communication. The internal consistency of the scale as measured by Cronbach’s alpha was 0.81.

Similarly, Table 3B presents the five items describing Professional Competencies. TSE/Es–mentor scores were uniformly high, and they uniformly rated the Scholars higher on these constructs than did the Scholars themselves, where the largest disparity was for the items describing the Scholars’ commitment to staff education. The internal consistency of the scale as measured by Cronbach’s alpha was 0.71.

Table 3C describes the characteristics for the 12 items comprising the Team Building and Team Sustainability scales. Here too, TSE/Es–mentor scores were higher, with the highest positive discrepancy for two items describing communication skills, one for the articulation of vision and goals and the second for communicating with clinical teammates. The internal consistency of the scale as measured by Cronbach’s alpha was 0.93.

Table 3D presents item characteristics for the six items representing the Appropriate Use of Resources and Execution of Study scale. The largest discrepancy between scores of TSE/Es mentors and Scholars was the item about delegation of responsibility. The internal consistency of the scale as measured by Cronbach’s alpha was 0.81.

Table 3E describes the five items comprising the Regulatory Accountability Scale. TSE/Es–mentors believed scholars were much stronger on hospital regulations and policy than did the scholars. The internal consistency of the scale as measured by Cronbach’s alpha was 0.97.

Table 5 presents the comments provided by scholars who completed the survey about their experiences in having the meeting with their mentor team and receiving the results of the evaluation. Overall, the comments reflected an appreciation for being exposed to these constructs in addition to the traditional clinical and research skills, and a recognition that these skills are key to a successful translational science career.

Table 5. Clinical scholar feedback after return of results

Discussion

Effective Team Science Leadership is vital to success of translational research, but few programs focus specifically on developing leadership skills in this area [3,11]. This paper describes our Team Science Leadership curriculum; the development of Team Science Leadership competencies for translational investigators leading multidisciplinary teams that develop and conduct human participant protocols; the value of having the Scholar, the Scholar’s mentor, and a diverse group of TSE/Es evaluate the Scholar’s progress in mastering the competencies; and the value of sharing the information with the Scholar for future career development planning. By having the Scholars rating themselves on entry and yearly thereafter and having the mentors and TSE/Es rating the Scholars at the end of each year in the program, we provide feedback in a timely way that allows the Scholars, mentors, and program leaders to individualize paths to attain the competencies by the time the Scholar graduates.

We converted the competencies into a semi-quantitative survey format and evaluated the psychometric characteristics of the survey responses. In general, the five resulting scales had high internal consistency and factor structure. Of note, evaluators tended to rate Scholars as being more competent across the range of leadership characteristics than the Scholars rated themselves. In reviewing the survey results with the Scholars, the leaders used these discrepancies to initiate conversations with the Scholars to better understand their perceptions and why they felt less confident about their competence than did the mentor and TSE/Es. This feedback was especially important in helping the Scholars develop self-confidence in their abilities. Overall, Scholars rated their participation favorably, commenting that it helped them appreciate better that Team Science Leadership skills are important in becoming successful translational scientists.

Our study has several limitations. First, while the survey was developed by people experienced in translational research, with input from and collaboration with experts from two other institutions, important competencies may be missing from the list or not optimally described. Moreover, our competencies focused on the leadership skills required to lead a clinical research protocol team, the central component of our training program, and thus may not address competencies required for research in other areas of the translational spectrum or at different points in a translational career. In addition, the survey was implemented at only our relatively small, basic science- focused institution. Therefore, both the results and Scholar experiences may be different at other institutions with a larger number of Scholars, with different Scholar recruitment and selection priorities, and different educational programs. In particular, some other KL-2 Scholar programs focus on junior faculty who are further along in their careers than our Scholars and so are already serving as team science leaders. To gain a broader perspective, we are currently collaborating with other academic institutions to gain additional experience on the generalizability of our findings. In addition, the factor structure, which supported single dimensions within scales, and the Cronbach’s alphas, which describe fairly high-scale reliability, will need to be replicated with a larger and more heterogeneous sample before drawing generalizable conclusions. For statistical and psychometric purposes, it would be optimal to have all evaluators score all Scholars on all dimensions. However, as expected, not all TSE/Es had sufficient exposure or knowledge of the Scholars to rate them on all dimensions (e.g., a TSE/E who interacted with a Scholar only on regulatory issues or budgets may not be able to evaluate the Scholar’s ability to express a vision), and so “missing” responses were common. Fourth, some TSE/Es did not complete all items on the survey, so it was not possible to discern whether this was due to an inability to rate or a failure to complete the survey. Finally, we consider the competencies a living document and so anticipate modifying the lists as we learn from our own experience and from collaborators at other institutions.

With these limitations in mind, the development, deployment, and results of the survey provide interesting information. First, the survey provides a starting point for discussing Team Science Leadership competencies for translational investigators who aspire to lead their own multidisciplinary groups. Second, the results of the survey provide information to Scholars based on the perceptions of their mentor and an experienced group of TSE/Es, nearly all of whom have interacted with numerous Scholars during their careers, and who observe the Scholars in a wide range of roles required for translational success. Third, the results of the survey inform the elements we include in our training program on a yearly basis. One ancillary benefit is that by participating in the process, and reviewing the competencies, the Scholars’ mentors undergo training in mentoring in Team Science Leadership.

Despite Scholars’ abundant experience in being members of clinical and scientific teams, few have formally thought of themselves as Team Science Leaders. That is why we have integrated the survey into a much fuller curriculum in Team Science Leadership to ensure that Scholars develop these competencies.

We try to emphasize that we do not expect Scholars to master all of the team science competencies rapidly, but rather want to help them identify the key skills and attributes needed and intentionally plot a multi-year path to achieve the goal that can be incorporated into the Scholar’s Individual Development Plan. We previously developed a Graduate Tracking Survey System (GTSS) [1,12] to assess the progress of our Scholars after graduation and it provides outcome data on our Scholars’ success as translational science team leaders. Over time, by merging the process-based assessments generated by the survey with the outcome data from the GTSS, we hope to gain a better understanding of the factors that are associated with Team Science Leadership success. That information will help us craft curricula and assessments to optimize our program. We are eager to share our survey with other institutions as we continue to refine and modify the competencies, survey, and curriculum.

Conclusion

Our Team Science Leadership initiative highlights the importance of defining Team Science Leadership competencies. Training Scholars in those competencies by embedding them in an experiential and didactic curriculum focused on clinical and translational science, measuring Scholars’ attainment of the competencies over time using a quantitative survey, and providing periodic feedback to Scholars with the focus on ensuring that the Scholar masters the competencies by the end of the program. We are eager to share our materials with educators at other institutions to help refine the competencies and to enhance external validity and continuous improvement, as well as to help systematize Team Science Leadership skills measurement and training in translational science across the CTSA network.