The innovation scorecard for continuous improvement applied to translational science

Abstract

Introduction

This paper reports on the baseline stage of a qualitative evaluation of the application of the Innovative Scorecard (ISC) to the Clinical and Translational Science Award (CTSA) at the University of Texas Medical Branch (UTMB) at Galveston. The ISC is adopted from the established Balanced Scorecard system for strategic planning and performance management. In formulating the evaluation, we focused on the organizational identity literature.

Methods

The initial evaluation consisted of a series of semi-structured interviews with 22 participants of the ISC Boot Camp conducted in July 2015.

Results

The logic of grounded theory pointed to the clustering of perceptions of the ISC around respondents’ occupational locations at UTMB. Administrators anticipate the expansion of planning activities to include a wider range of participants under the current CTSA award period (2015–2020) than under our first CTSA approval period (2009–2014). A common viewpoint among the senior scientists was that the scientific value of their work will continue to speak for itself without requiring the language of business. Junior scientists looked forward to the ISC’s emphasis on increasingly horizontal leadership that will give them more access to and more control over their work and resources. Postdocs and senior staff welcomed increased involvement in the total research process at UTMB.

Conclusion

The report concludes with strategies for future follow-up.

Objectives

The Institute for Translational Sciences (ITS) has been the major force behind the introduction and integration of the translational science movement at the University of Texas Medical Branch (UTMB) at Galveston. With an initial 5-year award from the NIH Clinical and Translational Science Award (CTSA) beginning in 2009 (herein known as CTSA 1.0), the ITS has designed clinical and research projects to enhance the effectiveness and practicality of its research agendas [1]. As is the case with all such major programs, the ITS instituted a sophisticated system of evaluation. The essential feature of this system was the introduction of logic modeling. In brief, logic modeling involves the projection of a team’s accomplishments and work tasks. Success is measured by means of traditional metrics [2].

With the renewal of its CTSA in 2015 (herein known as CTSA 2.0), granting an additional 5 years of support, the ITS administrators and evaluators sought a new management system as well as a sophisticated evaluation strategy. Issues to be overcome included tying each stakeholder’s actions to an agreed upon strategy (i.e., vision and mission based). The new system also had to provide time sensitive performance data that allows for lead and lag indicators. Finally, the new system had to provide actionable data conducive to rapid organizational change and adaptation.

The top management team in the ITS was committed to training a diverse workforce in the authentic skills needed to advance all phases of translational research. This included engaging stakeholders across all phases of translational research and clinical trials; integrating quality systems through all types of translational research, including clinical trials in special populations; and advancing the conduct of translational research through multidisciplinary translational team (MTT)-based innovation. The ITS has established MTTs, which are unique hybrid teams structured to include goals of both an academic research team in knowledge generation and training with those of a product-driven business team to develop a device or intervention for clinical translation. MTT design characteristics include a strategic core of multidisciplinary investigators dynamically engaged in training, capacity development, and product generation [3]. The interdependence and heterogeneous membership promotes innovation and effectiveness [4].

The Innovative Scorecard

The Innovative Scorecard (ISC) is adopted from the established Balanced Scorecard (BSC) system [5] for strategic planning and performance management system and it:

• ∙ Communicates with clarity an organization’s vision, mission, and strategy to employees and other stakeholders;

• ∙ Aligns day-to-day work to vision and strategy;

• ∙ Provides a framework for prioritizing programs, projects, services, products, and resources; and

• ∙ Uses strategic performance measures and targets to measure progress [6].

The established BSC has been adopted in numerous work settings, ranging from government to private industry [7]. Its application to biomedical research, however, has been limited, although it has been applied to the evaluation of healthcare delivery planning and delivery systems [8]. The ITS hypothesized the BSC could be practicably applied because of its ability to provide an overall strategy to focus upon; format the program’s experiences into a “story” to relate its accomplishments; and clarify what is specifically important to measure and evaluate. Contemporary uses of the BSC management system frequently focuses on “continuous improvement” in all areas to produce sustainable and growth-orientated performance [9].

UTMB’s ITS modified the BSC approach to strategic management to achieve 6 objectives. These are to:

• ∙ Align with UTMB’s “My Road Ahead” initiative which is an institutional vision that focuses on multiple constituents and performance measurements through scorecard analytics;

• ∙ Align ITS’ vision, mission, and strategy with the expressed focus and requirements of the National Institutes of Health/National Center for the Advancing Translational Science (NIH/NCATS) CTSA 2.0 philosophy and orientation;

• ∙ Articulate and operationalize the ITS’ approach to continuous improvement, which is mandated by the NIH/NCATS;

• ∙ Define a platform to evaluate performance and tell ITS’ “unique story” to internal and external constituents;

• ∙ Guide decision-making and resource allocation on the basis of clearly articulated objectives and initiatives that are related to vision and mission; and

• ∙ To create a common language to discuss strategy and operations that would facilitate a culture of improvement.

The ITS initiated a systematic rollout of its modified scorecard system for evaluating overall operations of the Institute in 2015. Shown in Fig. 1 are the 9 steps of implementation. To start this cycle, the ITS held a Scorecard Boot Camp to provide scorecard basics for 22 of its key personnel and leadership. Prior to this Boot Camp, ITS leaders worked with consultants on a comprehensive assessment involving an internal and external analysis; determination of strengths, weaknesses, opportunities, and threats; a customer value proposition; and creation of a new ITS vision and mission. Breakout groups were responsible for developing basic themes and ITS strategic themes, which were then used to develop 14 strategic objectives and a preliminary strategy map.

Fig. 1 The 9 Steps of a balances scorecard system (From Rohm et al. [6]).

The introduction of the ISC also called for an innovative evaluation strategy. This paper reports on the initial stage of a qualitative approach to evaluating the ISC in the ITS at UTMB. In formulating a strategy for our evaluation, we focus on the literature on organizational identity, especially in terms of members’ perceptions of change and resilience of the organizations’ identity as well as the impact of perceived change on members’ self-identity.

Organizational Identity: Change and Resilience

The systematic study of organizations can follow 1 of 2 directions. On the one hand, the researcher can attempt to describe and explain the organization as an almost material-like entity. This approach was most common in the traditional and early study of organizations as systems [10]. The machine-like imagery of the organization led to the search for metrics to rationally and mathematically replicate the actual organization. The more recent approach has been to study organizational identity. This approach emerged with the growing understanding that organizations are not things, but rather images, concepts, ideas, and perceptions of the way people—both insiders and outsiders—think about the organization in which they work or with whom they interact.

Gioia et al. reviewed the extensive and growing literature on organizational identity change that has accumulated over the past 30 years or so [11]. They begin with the classic definition of organizational identity as “those features of an organization that in the eyes of its members are central to the organization’s character or ‘self-image,’ make the organization distinctive from other similar organizations, and are viewed as having continuity over time.” They then typed the conceptual and empirical work into 3 commonly accepted “pillars” of organizational identity: central, distinctive, and enduring [12].

Central features of organizational identity include key values, labels, products, services, and practices. They are deemed to be essential aspects of organizational self-definition. A viable history of an organization is a necessary component to figuring out whether it is currently performing consistently with its founding or adopted core values. Gioia et al. note that of the 3 definitional pillars noted above, central features may be the most essential ones, simply because if there are no perceived central or core features, then it is difficult to even conceive of the idea of identity [11].

Distinctive features distinguish their concept of the organization from other organizations. The notion of distinction is a key assumption of identity at any level. A possible paradox lies in the desire to be distinctive from others yet communal enough that all members identify to a single major category—industry, market, research sector, etc.

Enduring features are perhaps the most important to the current study of the application of the ISC to translational science. The question is one of whether an organization can maintain a semblance of continuity in the face of change. Gioia et al. and Albert and Whetten agree that the now substantial body of research affirms that identity often changes over relatively short-time frames, albeit perhaps in subtle ways [11, 13]. Members tend to perceive identity as stable, even when it is changing, because they continue to use the same labels even though meanings of those labels may change without awareness.

The third pillar is especially relevant to our study because it is home to and conducive to our sociologically-informed interest in organizational culture [14]. Organizational culture consists of the most common, taken-for-granted ways of perceiving and doing things. Accordingly, we argue that changes in organizational identity, especially the subtle and taken-for-granted changes, will reflexively impact the way members do things, say things, categorize things and feel about things on an everyday life basis.

Typologizing Members’ Perceptions and Responses to Change

An important aspect of studying organizational change is to see it through the everyday experiences and perceptions of members. Dibella argues, on a common-sense level, that there are 2 dimensions of organizational change that shape members’ response to change factors: appeal and likelihood [15]. These 2 factors point to personal, often nonprofessional, rarely scientific but always sensible ways to make sense of change.

Dibella notes that the key perception or cognition is whether some planned or intentional change is considered desirable or undesirable. This distinction is a key factor for a member of an organization to decide whether to engage productively in a change initiative. This perception provides a foundation upon which resistance or participation ultimately rests. Anyone who views change as undesirable is unlikely to help bring it about or, even worse, may sabotage the efforts of those trying to make a desirable change.

The second distinction for change is the likelihood that the change will be realized. Certain changes are inevitable; others are impossible, or regarded as such. Dibella gives the example of the aging process and all the inevitable changes that accompany. Other inevitable changes include market maturity, increasingly sophisticated customers, improved technology, and turnover staff.

Four scenarios are possible as the 2 factors interact. When change is perceived as both desirable and inevitable, the member feels encouraged to expedite the change. When the change is perceived as desirable, yet impossible, the member is dependent on the manager’s willingness and ability to increase conviction that the change is possible. When the change is perceived as not very desirable, yet with a high likelihood of occurring, the member is dependent on the manager to redefine the change as positive. When the change is perceived as undesirable, with a low likelihood of happening, the manager faces the retest challenge to implementing change [16]. Although Dibella’s 2×2 model may appear a bit oversimplified, it very clearly illustrates the value of studying organizational change—in terms of identity or culture—through a social constructivist perspective and as experienced by the member. The model also successfully demonstrates the factors feeding into the experience of organizational change can be many, varied and complicated.

The present study is inspired by the above ideas from the literature on organizational change. In addition, NCATS has strongly called for mixed qualitative and quantitative methods in designing evaluation protocols, as well as piloting new evaluation methods [17]. We believe the study described below meets these recommendations.

Methods

The goal of the initial stage of this qualitative process evaluation was to establish a baseline data set upon which to assess organizational and cultural change with the ISC longitudinally [18]. The Institutional Review Board at UTMB reviewed and approved this research project. We collected data by means of semi-structured interviews with the BSC consultants, CTSA leadership and key staff participating in the first ISC activity, and a Boot Camp held in July 2015 (n=22). The interviews themselves transpired from September 2015 through August 2016. This period of time encompassed the first 4 stages of scorecard implementation shown in Fig. 1. The first author conducted all the interviews at times and locations determined by the respondents. The interviews were audio-recorded and transcribed. Specific interview items included, among others:

• ∙ Participants’ preliminary expectations of the ISC and the Boot Camp;

• ∙ Participants’ understanding of the objectives of the ISC;

• ∙ Participants’ early thoughts on the fit between the ISC and their ongoing scientific work;

• ∙ Participants’ expectations of the impact of the ISC on their sense of membership at CTSA/UTMB;

• ∙ Examples of recent, actual events/behaviors/decision-making that might be related to the ISC; and

• ∙ How, when, and with whom the ISC is discussed in normal conversation at work.

The 22 respondents represent the majority of Boot Camp participants. During interview, however, they were asked to speak for their colleagues who had expectations of the BSC but were unable to attend the Boot Camp.

Early Perceptions of the Innovation Scorecard

Following the inductively oriented, grounded theory method of qualitative analysis [19], we found that respondents’ perceptions were clustered in terms of their positions in the organization and their status on the MTTs. This was to be expected, given the Boot Camp’s facilitators’ efforts to clarify specific aspects of training by features relevant to diverse team members. Our preliminary analysis provides 5 clusters of respondents’ positive expectations of the ISC system and concerns for its implementation. We present some of the more interesting preliminary findings, not necessarily general to all respondents in each category, but highly suggestive for the design and execution of ongoing evaluation.

Administrators/External Advisory Board

Expectations

These respondents (n=4) in general perceived the ISC to be a rational and user-friendly system of strategic planning, management and evaluation, as presented to them by the BSC consultants and the ITS evaluators at the Boot Camp and beyond. ISC seemed to fit well with the parameters of healthcare and research settings. Perhaps the most highly anticipated feature of the ISC was the expectation that planning activities could now be extended to most CTSA 2.0 participants. These would include all levels of administration, faculty, and staff. This expectation contrasted with CTSA 1.0 strategic planning, management and evaluation (e.g., logic models) for which responsibility for planning appeared to be limited to the top administrators. The CTSA 1.0 system resulted in added work to the administrators and board members that would eventually be delegated to lower status participants (e.g., staff, postdoctoral students).

Concerns

The administrators’ and board members’ primary concern was the additional resources and funding the ISC would require. The ISC required all team members to be involved upfront in the planning and evaluation processes. It also resulted in higher clinical and scientific expectations. The administrators’ and board members’ concern now is that the required additional resources and funding from external sources to fund these new activities will remain problematic.

Senior Scientists/Professors

Expectations

Senior scientists and professors (n=6) in general felt that the ISC system will support the ongoing emphasis on team science in the CTSA at UTMB. Since the inception of the CTSA at UTMB in 2009, these respondents have generally bought into the serious emphasis on team science which was established as a program priority by the CTSA leadership [20]. Put differently, the team philosophy appears to fit very well with the ISC.

Concerns

The senior scientists and professors, who represent the bulk of the principal investigators affiliated with the ITS, are in general not convinced of the wisdom of introducing an inherently business model of management and planning like ISC to an academic research setting. The vocabularies of ISC and its parent BSC still seem a bit out-of-place in the researchers’ labs, with verbiage such as “entrepreneurship” and “enterprise” being used. The most general feeling among the scientists is that the organizational culture at UTMB still nourishes the belief that the inherent scientific value of their work there will speak for itself and be communicated best through scientific channels.

Junior Scientists/Professors

Expectations

Junior scientists and lower status professors (n=6) in general look forward to the ISC. They anticipate positively the increasingly horizontal leadership structure that will give them more access to and therefore more control over their careers, their work, and available resources.

Concerns

The junior scientists and professors perceive a paradox in expectations on their work. There is a general fear that the increasing emphasis on collective planning and management activities may be contraindicated by continuing organizational emphasis on individualistic values and procedures for tenure, promotion, and merit rewards. A related concern is stress placed upon time and effort allocation. A common way to posit this concern in interview is to cite the “100% time” dilemma. The junior scientists and professors understandably argue that they are already working at or above capacity, and their ability to allocate increased tasks to subordinates is much less than that of their seniors.

Postdocs and Graduate Students

Expectations

Postdoctoral students and graduate students (n=4) in general expect increased involvement in, serious responsibility for, and experience in the total research process at UTMB. They feel that their traditional or current positions are overly focused, for example, in terms of research assistance. They anticipate a wider range of tasks and responsibilities, especially in terms of research design and community involvement.

Concerns

The postdoctoral students and graduate students maintain the strong hope that senior leadership will not operationalize ISC goals and activities as tasks to be relegated to students and other subordinates. Team management is a strong concern. Most essentially, the students voice the traditional concern over adequate funding for their positions.

Senior Staff Members

Senior staff members (n=4) look forward to increasing team and team-like group activities within which to conduct their activities. Their optimism was seeded in the Boot Camp meetings that were populated by ITS personnel at all levels. These sophisticated group activities would hopefully reduce the occasional feeling of isolation from the team/translational science philosophy experienced in CTSA 1.0.

Concerns

A general concern is that the tasks given to their newly formed staff teams may not always be consequential and meaningful. “Important” tasks may still be relegated to individual staff members. This concern is tempered by staff’s insistence that they hold considerable responsibility for demonstrating their ability to perform high-order and consequential tasks.

Discussion

The data strongly suggest that the success of the ISC rests heavily on participants’ ability to integrate the ISC into everyday culture at UTMB [18] and address resistance through effective institutional politics [20]. The previous approach of introducing a management/evaluation strategy separate from the everyday culture at UTMB relegated it to the status of a task that was too easily assigned to lower status research team members (e.g., postdocs and graduate students). There is an overall positive feeling among all levels of participants that the above goal can be achieved, given the innovative system of teams developed at UTMB to structure research activities [21].

The exploratory nature of this study led to the discovery of the use of “paradox” in the introduction of the ISC in the ITS at UTMB. The idea of paradox has become increasingly visible and important in the literature on organizational studies. As Lewis notes [22], paradox across disciplines generally refers to thing-like perceptions and situations in organizations marked by contradictions, inner conflicts, and/or unexpected surprises. In terms of organizational studies, paradox can be seen as a way managers and researchers make sense of the inconsistencies in the groups they work with or observe. Of direct relevance to the social constructionist orientation of this study, however, is the way respondents use the idea of paradox to make sense of their positions in the BSC. Watzlawick et al. [23] use the concept transcendence to describe actors’ ability to think paradoxically. In this present study, respondents use the term paradox on 5 different occasions. This finding may not appear significant in its own right but it may represent a distinct progression in the way scientists conceptualize and talk about the ISC. We argue that thinking of their work problems in rational terms like “paradox” marks a more constructive way of dealing with change in the structure of science than terms used in the early days of the translational science movement such as “bureaucratic craziness” and “trying to make science a business.” This shift in conceptualization and talk should be encouraged.

The second stage of the qualitative evaluation of the ISC at UTMB will consist of naturalistic observations of meetings, lab activities, training sessions, and other occasions to code ways the BSC might be emerging as part of normal interaction and culture. The first stage of this evaluation research suggests the following items to be monitored:

• ∙ Changes in the semantics of everyday talk at the ITS at UTMB. We would expect a continuing increase in rational and interdisciplinary vocabulary;

• ∙ The progress of the newly-established staff teams, especially in terms of the processes by which leadership evolves or is imposed;

• ∙ The actual restructuring of research teams to include nonscientific and technical members such as patent attorneys and bioinformatics experts;

• ∙ The redirection of ancillary services such as ethics and community involvement to fit the new system;

• ∙ The system of monitoring efforts at strategic planning, especially participants’ perceptions of these efforts; and

• ∙ The impact of ongoing organizational change on the self-identity of the scientists.