Normalizing Diversity in Merit Review Forms

Following the protests surrounding the George Floyd killing, the chair of the Government Department of the University of Texas at Austin created a diversity taskforce. It had two purposes: (1) identify the challenges facing the department with respect to diversity and inclusion and (2) propose recommendations to enhance the study of race and ethnic politics in the department. The taskforce—comprising two graduate students, one staff, and eight faculty members—met weekly and considered a set of challenges and recommendations. Two themes appeared repeatedly in our meetings. One is that faculty lacked a channel for informing others of their activities to promote diversity in research, teaching, and service. The absence of transparency about these activities is particularly problematic because women and faculty of color typically engage in them at disproportionate rates (for a discussion of workload inequities, see O’Meara et al. 2021). The other theme is that faculty are not being incentivized to promote diversity and inclusion. Diversity and inclusion efforts often have costs: they require considerable investments of time and resources by faculty. Yet, they matter for academia and society in general; as such, departments and universities should recognize and reward faculty who engage in these efforts. In this article, we propose one way of doing so that focuses on merit review forms.

MERIT REVIEW FORMS ACROSS R1 UNIVERSITIES

In the Government Department of the University of Texas at Austin, every faculty member must submit a “Faculty Performance Summary” form each year. The merit review form details what each faculty member has accomplished in the calendar year. A committee—comprising 12 individuals—collectively discusses and rates whether a faculty member exceeds, meets, or is below expectations. The same committee also uses the performance summary to determine merit raises. Regardless of whether there is a raise pool, this process prioritizes the listed categories (e.g., research, teaching, and service)—and thus gives no attention to what is not listed (although faculty may add other information).

The University of Texas at Austin is certainly not the only university to use merit review forms. We reached out to individuals at the top 101 R1 institutions per US News and World Report ¹ and asked for copies of their forms. We ended up with responses, copies, or both from all 101 institutions. Three differences caught our attention.

First, institutions vary in who administers the forms. In some cases, it is just the individual chair or the department; in other cases, the forms are administered by either the specific college or the entire university (table 1).

Table 1 Distribution of the Unit Administering Merit Review Forms

The second difference relates to the when; that is, how often faculty members fill out the forms. Although the modal frequency is annual (85%), one SEC institution requires them every semester. The remaining 15 institutions (the majority of which are in California) only ask for the forms every two or three years, often coinciding with promotion reviews.

The third difference is how the structure of the forms differs across institutions. At one end, there are some institutions (N = 6), most of which are private, that ask faculty members to submit just their CV. At the other end, there are institutions that have highly “bureaucratized”² and “infernal”³ systems with a “zillion,”⁴ “obnoxious”⁵ number of (sub)categories.

What struck us even more, however, is the uniformity of the categories (table 2). Every institution has a research section. All but one asks about teaching. Interestingly, only half the institutions (N = 51) placed research before teaching. Ninety-nine institutions have at least one service section. And almost every form includes grants, although the vast majority subsume it under the “research” category. Thirty institutions have a standalone category for external funding.

Table 2 Categories on Merit Review Forms

¹ Explicit standalone section.

² Inclusive of explicit standalone sections and subsections.

However, only 15 institutions ask about diversity. Put differently, twice as many institutions explicitly ask about grants than about diversity in any way. Syracuse University, for example, is one of the few institutions to have a standalone section on diversity: “Contributions to Equity, Diversity, Inclusion, and Internationalization” (figure 1). Alternatively, some institutions ask about diversity as subsections of existing sections. University of California Irvine, for example, asks about “Teaching Activity” (figure 2), and within this section, there are subsections including “Diversity Activities Related to Teaching.” We see the same structure in the form’s other sections, such as “Research” and “Creative Activity.” Finally, there are other institutions—for example, the University of North Carolina—that have an “Additional Information” section that pointedly asks faculty to discuss their efforts to promote diversity and inclusion (figure 3).

However, only 15 institutions ask about diversity. Put differently, twice as many institutions explicitly ask about grants than about diversity in any way.

Figure 1 Example of Standalone Diversity Section

(Syracuse University)

Figure 2 Example of Subsumed Diversity Subsections

(University of California Irvine)

Figure 3 Example of Diversity as Additional/Optional Section

(University of North Carolina)

A few disclaimers warrant mention. First, we assumed the forms would be constant across individuals. Therefore, each of us reached out to individuals whom we knew at each institution. Barring a few exceptions, we asked one individual per institution. The characteristics of the individual providing the forms varied substantially in gender, race or ethnicity, and rank. This is relevant for our second point. Many institutions had online systems with preassigned categories. If a faculty member did not have any data relevant for that category (e.g., patents), it would be left blank, and the subsequent PDF-generated form would omit this category. Thus, the copy we received would have no indication of said category. Without inquiring, we would not be able to infer whether a category was missing in general or because the individual had nothing to claim in that year. We saw evidence of this when a senior faculty member listed no teaching because of having a high administrative service load. We also saw this with a female faculty member who did not teach because of maternity leave. In both cases, there was no doubt that these two institutions had a teaching category (which was confirmed in emails).

The category of concern for us is the one for “diversity.” It is possible that an institution has a diversity section, but the individual left it blank. Although we may end up undercounting which institutions have diversity sections, we cannot imagine the undercount being substantial. When soliciting the forms, we were explicit that our research effort was about diversity. It should have primed our contacts to acknowledge diversity in their emails. Moreover, not all forms were generated from online systems that dropped categories. The ratio of online forms with diversity sections versus without (whether explicitly or dropped) those sections is statistically nondifferentiable from all other form types (p = 0.13).⁶

Having established the categories on these merit review forms, we call attention to the structure of these categories. We know from the literature on censuses that the choice of which categories appear on the form is political (Anderson and Fienberg 1999; Csata, Hlatky, and Liu 2021; Parvini and Simani 2019). In some contexts, when a category is left off a form, it suggests it is of lesser (or no) interest and value. And when this omission occurs repeatedly, it can become a self-fulfilling prophecy where the general population forgets or is simply unaware that the omitted category even exists. Even if the population at large is informed and supportive, there is always the risk of the “out-of-sight, out-of-mind” phenomenon.

As a discipline, we talk about the importance of diversity— from graduate students to faculty, from bibliographies to syllabi. We even increasingly ask for diversity statements from job applicants. Yet, diversity continues to be absent in our merit review forms—whether as a standalone category (e.g., “diversity”) or as a subcategory of an existing category (e.g., “diversity in teaching”). Given the importance of these forms, this omission is concerning. Yet, we also contend that changing these forms provides a concrete opportunity for promoting diversity-related activities in the profession.

TWO RECOMMENDATIONS FOR MERIT REVIEW FORMS

Inclusion in Merit Review Forms

When merit review forms include a specific (sub)section, it serves two purposes. First, it informs; that is, it makes something transparent (O’Meara et al. 2021). It provides an opportunity for the evaluated faculty members to identify their accomplishments in that category. It spotlights what they have done (or, glaringly, what they have not done). Moreover, when everyone is asked to provide this information, evaluators are better positioned to (1) evaluate the concerned issue and (2) compare across faculty members. Consider research. Although there are variations in how the category is subcategorized—for example, books versus articles or solo authored versus coauthored—the existence of this category allows faculty to signal their research productivity.

Second, when merit review forms include a specific (sub)section, it incentivizes, suggesting to the evaluated faculty that the contents of that section deserve credit (O’Meara et al. 2021). It is not just about whether the contents matter but also how much they matter. And if these are the evaluating criteria, faculty members may respond by doing things that fall in that (sub)section and by doing them well. They may also reward colleagues who undertake these efforts. Consider teaching. Faculty members are evaluated not only on whether they teach but also on how well they teach and what they have done to improve their teaching. There are teaching awards, there are course development funds, and there are teaching mentorship programs. Reporting these awards, funds, and programs explicitly on merit review forms incentivizes faculty to be better teachers.

Given these considerations, we recommend one of two strategies. We are agnostic as to which strategy is more effective because we do not have the empirical evidence. We do, however, have strong priors that doing either one is better than doing neither.

The first strategy is to use merit review forms that explicitly include a section on contributions to diversity, as Syracuse University does. This section would be a place where faculty can signal effort and provide context. These contributions could include diversifying syllabi, advising organizations for underrepresented groups such as Alpha Kappa Alpha, and participating in APSA’s Diversity and Inclusion Hackathon. We believe giving diversity its own standalone category—on the same level as research, teaching, and service—sends an important message about how the department values inclusion. We are, however, also cognizant that this may be the more difficult strategy to adopt because of institutional constraints (e.g., merit forms are designed by the higher-ups) or ideological proclivities of department colleagues (both are discussed later). One objection to this approach might be that it leads to double-counting: the same activity may be listed in two sections. We would note, however, that substantial double-counting may already take place, given that none of the forms explicitly prohibit it. For example, a faculty member may list an article published with a graduate student in both the research and teaching sections. Moreover, we believe that double-counting would be a good thing if it led departments to provide greater rewards for efforts to promote diversity and inclusion.

The second strategy is to create diversity subsections within each of the major sections. This is what the University of California Irvine does. Doing so would not only identify the research, teaching, and service someone did but also how it contributed to diversity efforts. Employing such a strategy would eliminate the double-counting problem while still pushing departments to recognize and reward diversity-related activities. Using subsections would also call attention to the differences among various types of service, teaching, and research. Just as there are differences in courses taught and committee assignments, there are also factors that make diversifying syllabi, advising organizations for underrepresented groups, and participating in APSA’s inclusion taskforce efforts qualitatively different from other types of teaching, university service, and professional service. The one downside to using this approach is that it might make the diversity-related activities less salient by splitting them across multiple categories.

Placement in Merit Review Forms

We also recommend that the diversity section be placed at the top of the merit review form or at the beginning of its respective subsections. It is not enough for merit review forms to include a (sub)section on diversity. Adding a diversity section at the bottom of the form or at the end of each section risks signaling it is a mere afterthought. Additionally, we know from the literature on survey designs (Ben-Nun 2008; Tourangeau, Rips, and Rasinski 2000) that fatigue can be a real problem in answering questionnaires. Respondents are less likely to complete the last parts of a survey; even if they do, responses to the last questions tend to be cursory and less well thought out. Having a diversity section at the very end (or at the end of each section) risks the same fatigue. Additionally, there is the risk of fatigue on the part of the evaluators. When evaluators have to read through multiple merit review forms, they can end up skipping large chunks at the bottom.

We also know from the survey design literature that the first set of questions can prime respondents, affecting how they respond to subsequent questions (Gaines, Kuklinski, and Quirk 2007; Parkin 2008). In the same vein, putting a diversity section at the top of merit review forms can prime how evaluators assess subsequent information. It highlights—to continue the earlier examples—that the faculty member did not just update a syllabus but also diversified it by incorporating minority voices. Likewise, it calls attention to how the faculty member did not just serve as a faculty adviser to a student organization but also advised an organization serving underrepresented students. Highlighting faculty members’ diversity-related efforts at the outset not only calls attention to what they have done for diversity but also underscores everything else they have done, despite all the time they spent on efforts to address inequity.

Caveats

We have assumed that departments have the authority to change the categories listed on these forms and their placement, but this may not always be the case. Under these circumstances, departments have two non-mutually exclusive alternatives. The first solution—one used by a Texas-based institution—is for departments to adopt an additional form at the department level. This may be effective because departments are often the unit that determines merit raises. The second solution—one put in place by an East Coast institution—is for the department chair to bring the issue to the higher-ups to effect change. This is certainly time consuming, but normalizing diversity requires sustained and substantial lobbying efforts.

We have also assumed that members of the evaluating body want to address inequity; that is, interest > 0. The challenge is therefore to create the medium for transmitting information and the structure of incentives to act. However, it is possible that members of the evaluating body do not care about diversity: interest = 0. Under such conditions, having and highlighting a diversity section may have a countervailing effect. It may, in fact, prime members of the evaluating body to be resentful when evaluating the other sections. This is a challenge we are not positioned to address; instead, we choose to focus on those who do care about diversity.

We know from the representation literature that just because there are women and minorities in Congress does not mean we will see the adoption of more women- and minority-friendly policies—for a host of reasons including “silencing, stereotyping, enforced invisibility” (Hawkesworth 2003, 526). Likewise, even if the diversity section is at the very top of the merit review form and in boldface, the evaluating body may choose not to reward it if certain norms are absent (see O’Meara et al. 2021). Having said that, we are cautiously optimistic that revising the merit review forms can be a crucial first step in addressing important issues about workload balance, inclusion efforts, and diversity in the discipline.

DISCUSSION

In our diversity taskforce meetings, we noted two themes. First, faculty were not being informed about what other colleagues, especially faculty of color, were doing to increase diversity, nor were faculty of color being given a proper venue to highlight what they were doing. Second, faculty were not being incentivized to engage in or reward diversity-related efforts. Including a diversity section at the top of the merit review forms or in their respective subsections (e.g., research, teaching, and service) makes information about diversity-related efforts transparent, thereby providing the necessary credits to incentivize faculty to address inequity. Given that all departments undergo regularized review processes, we expect recommendations to have broad applicability. Moreover, given the systemic challenges to diversity, it is critically important that the discipline take seriously the incentive systems in place.