Citations to the Publications of Male and Female Political Scientists Revisited

Abstract

Prior research finds that women earn fewer citations than men for their publications, and it offers various reasons why this is the case. This study provides new evidence on these citation differences from two datasets on career citations earned by male and female political scientists. Our findings extend and elaborate on those in earlier research. Most notably, we find that older cohorts of women demonstrate substantial progress toward citation equity with their male peers.

Considerable research demonstrates that, on average, female political scientists earn fewer citations to their journal-article publications than male political scientists (e.g., Dion, Sumner, and Mitchell 2018; Maliniak, Powers, and Walter 2013; Mitchell, Lange, and Brus 2013). Two reasons for this disparity are that men are more numerous in the profession and that they are less likely than women to cite articles authored by women. Dion, Sumner, and Michell (2018, 314–17) summarized several studies for other disciplines that produced similar results.

These findings raise concerns about how scholarly authors choose works to cite, how the research records of male and female scholars are portrayed in citation data, and whether there is “equity” or gender bias in citation data. However, the authors of the previously cited publications call for more research to explain better the underlying character and causes of gender-based citation differences. This article provides new evidence on these differences using an alternative research design and data sources.

The research cited previously cleverly exploits data in which journal articles are the units of analysis and in which analyses of the gender of the authors and other attributes of articles reveal various citation patterns. An alternative analytic approach provides complementary and elaborating information using individual scholars as the units of analysis and assessing their career success in research recognition. The latter approach supports replication tests of some previously tested hypotheses, tests of logically compatible hypotheses, and tests of novel hypotheses that are not testable when articles are the units of analysis.

EXPECTATIONS FOR OVER-THE-CAREER CITATION COUNTS BY GENDER

The most elementary implication of the research cited previously, and Hypothesis 1 for this article, is that if individual articles written by women earn fewer citations than those written by men, then individual scholars’ overall citation records should demonstrate the same gender disparity. Considerable research on scholars as units of analysis provides support for this expectation, as well as for a comparable gender bias in numbers of articles published (e.g., Djupe et al. 2020). However, there is no comparably strong evidence in existing research for the following, more discriminating expectations.

The most elementary implication of the research cited previously, and Hypothesis 1 for this article, is that if individual articles written by women earn fewer citations than those written by men, then individual scholars’ overall citation records should demonstrate the same gender disparity.

Consider Maliniak, Powers, and Walter’s (2013, 890) unexpected finding that early-career women are especially under-cited compared to early-career men. Yet, this pattern may be sustained over time. Teele and Thelen (2017, 437) demonstrated two notable gender-by-rank gaps in publishing in 10 leading political science journals. Female assistant professors publish considerably fewer papers in these journals compared to their number in the profession than male assistant professors, but this disparity is the same at the rank of professor. Because men publish more articles in these journals during their career, their potential for earning more citations than women may be sustained throughout their career.

Two hypotheses about gender disparities are implied by the preceding evidence. Hypothesis 2 is a replication test of the Maliniak, Powers, and Walter (2013) finding with different data: that is, younger male scholars demonstrate especially higher citations than female scholars with comparable time in the profession. Hypothesis 3, also implied by the preceding evidence, has not been tested with data on articles: the gender gap in citations should increase over time in scholarly careers because gender disparities in publishing are sustained over time.

Mitchell, Lange, and Brus (2013, 490) and Dion, Sumner, and Mitchell (2018, 316) also observed that most scholarly disciplines in which women are numerous demonstrate less gender bias in citations. Yet, Dion, Sumner, and Mitchell (2018, 321–22) provided evidence that “greater gender representation in an academic field” improves the citation rate for articles written by women, but it does not eliminate the citation gap because men are less likely to cite scholarship by women in all fields. Our compatible Hypothesis 4 for individual scholars’ careers is that the gender gap in citations earned by women in fields in which they are especially numerous should be lower than those in fields in which they are less numerous. Another compatible expectation, however, is Hypothesis 5, that greater female presence in a field will not eliminate gender bias in career citations.

Maliniak, Powers, and Walter (2013, 899) also demonstrated that articles published by authors at R1 institutions earn higher citation counts. However, they did not investigate how that advantage relates to the gender gap in citations or propose a hypothesis about it. Our study provides evidence on whether women and men employed in prestigious departments enjoy the same citation benefits.

Finally, we extended the findings of prior research in another way. The research cited previously considers only journal-article citations. Yet, many researchers doubt whether article citations adequately represent the scholarly work of political scientists. Some prefer the broader scope of Google Scholar citation data, especially because it captures citations to books and book chapters (e.g., Jensenius et al. 2018, 821). Thus, we replicated tests of our hypotheses with both the Social Science Citation Index (SSCI) for journal articles and Google Scholar citation data.

THE DATA FOR OUR HYPOTHESIS TESTS

Our first dataset was created by Masuoka, Grofman, and Feld (2007a) for faculty members of US PhD-granting departments in 2002 to identify the “Top 400” cited scholars. The citation data are from the SSCI for 1960–2005. Thus, these citations encompass a larger set of journals than the studies of gender bias in citations described previously, and the temporal span of these SSCI data is much longer. These attributes are especially valuable for representing the research of political scientists in diverse subfields as well as those in age cohorts that have been long in the profession. We acquired this dataset from the Harvard Dataverse; details are described in the online appendix (Hill 2024).

Our second dataset is Kim and Grofman’s (2019a) for the Google Scholar citations of tenure-track faculty in US PhD-granting departments in 2017. It includes citation data for 1960–2017. This dataset was provided by Professor Hannah Kim and is described in more detail in the online appendix.

Both datasets include measures for total citations, gender, years in which PhDs were earned, institutions at which PhDs were earned, research fields, and institutions where scholars were on the faculty in 2002 and 2017, respectively.

We also investigated how the prestige of the employing institutions affects citations, which Maliniak, Powers, and Walter (2013) modeled with an R1 dummy variable. We adopted a measure of departmental prestige compatible with the Top 400 focus of the two datasets that we analyzed. Thus, we added to the first dataset a dummy variable for top-ranked departments using the first criterion in Masuoka, Grofman, and Feld (2007b): for those having produced the largest numbers of Top 400 graduates. We selected the following 11 departments that graduated 10 or more such scholars by citations for this dataset and that collectively account for 268 of the Top 400: Harvard (54), Yale (46), Berkeley (30), Michigan (27), Chicago (25), Princeton (19), Columbia (17), Stanford (16), North Carolina–Chapel Hill (12), Wisconsin (11), and MIT (11).

For the second dataset, we created another dummy variable for top departments using the criterion in Masuoka, Grofman, and Feld (2007b) for those departments that graduated notably substantial numbers of Top 400 scholars by Google Scholar rankings. We chose the following 11 departments that graduated 10 or more of the Top 400 cited PhD scholars who accounted for 252 of the Top 400: Harvard (51), Berkeley (38), Yale (30), Michigan (27), Chicago (22), Stanford (21), Columbia (16), Princeton (15), Rochester (11), Wisconsin–Madison (11), and Northwestern (10).

We also replicated the following analyses with a second measure of that concept from Masuoka, Grofman, and Feld (2007b) for those departments whose faculty collectively had especially numerous average research citations. The replication tests produced patterns of gender disparity in citations that closely match those reported in this study.

HYPOTHESIS TESTS FOR THE 2002 FACULTY USING SSCI CITATION DATA

Conventional multivariate models for our two datasets demonstrated that time in the profession, serving in a top department, and male gender are positively associated with more citations. These models are reported in detail in the online appendix. Yet, Achen (2002, 441–44) cautioned that such analyses with even this small number of predictors cannot be interpreted to mean that time in the career and a top department position have the same effects for men and women. He recommended examining cross-tabulation tables to assess the latter possibility. Our empirical analyses adopted this recommendation.

Table 1 reports empirical evidence from our first dataset of the type that Achen (2002) called for—with mean citation counts for various subsets of faculty within four quartiles by the years in which they earned their PhD—and provides compelling support for two of our first three hypotheses. Hypothesis 1, the common expectation that men will earn more citations than women, is supported in 11 of 12 equivalent paired comparisons (i.e., for men versus women in the youngest age quartile who teach in a top department). The exception is for the oldest quartile of faculty not teaching in a top department in which the female mean is modestly larger than the male mean but is not significantly higher by a t-test. Overall, however, the first hypothesis receives strong support.

Table 1 Social Science Citation Index Citations for Political Scientists in PhD-Granting Departments in 2002 by Year-of-PhD Quartiles, Teaching in a Top Department or Not, and Gender^(a)

Notes:

^(a) Cell entries are the mean number of SSCI citations for the subgroup, the N of cases, the range in parentheses, and a t-test in brackets for difference in means when women in a matched male–female pair have a higher mean than men.

^(b) Numbers in the quartiles differ because of tied cases at the cut points.

Source: Masuoka, Grofman, and Feld (2007a)

Hypothesis 2, that younger men especially will earn more citations than younger women, also is strongly supported. The gender gap for all of the youngest male–female cohort pairs is the most significant by far in percentage terms compared to every other paired male–female comparison in the same pair of columns. However, Hypothesis 3, that the gender gap will increase over time in scholarly careers, is not supported. In every paired comparison except one, the gender gap decreases markedly for women in the older cohorts. Only the citations for the set of five women in top departments and the oldest cohort do not follow this progression—perhaps because of the small number of women in that cohort. Yet, the younger three cohorts for top-department women follow the trend in other pairings. Thus, despite evidence that men have an advantage in publications and citations early in their career, have been shown to publish more at all ages in some analyses, and include in their numbers notable outliers with high citation counts in some subsets, older cohorts of women demonstrate increasing citation parity despite these circumstances.

However, Hypothesis 3, that the gender gap will increase over time in scholarly careers, is not supported. In every paired comparison except one, the gender gap decreases markedly for women in the older cohorts.

Finally, women in top departments, as the men there, are cited more frequently than women not in top departments. However, women in top departments are not more competitive with their temporal male peers in comparative citation rates than other female faculty.

Because of space limitations, a table with tests for Hypotheses 4 and 5 for 2002 about gender differences across research fields based on the representation of women there is presented in the online appendix. The percentages of women researching in each field, however, generally are low, ranging from 17% to only 23%. Thus, the prospects for the presence of women affecting citation rates for other women appear to be low, and the male means are significantly higher than the female means in all of the fields. The two fields with the lowest representation of women—international relations and methods—have the highest disparity in female-to-male citations, but the other fields do not demonstrate a general pattern. However, political theory and public administration/policy/public law have relatively high female-to-male citation ratios despite having only moderate female representation. Perhaps these fields have distinctive norms about citations regardless of the gender of the author. Overall, however, there is insufficient evidence to support Hypothesis 4, that female representation in a field reduces citation disparities by gender. Without evidence in support of Hypothesis 4, Hypothesis 5 is moot.

HYPOTHESIS TESTS FOR THE 2017 FACULTY USING GOOGLE SCHOLAR DATA

Table 2 presents evidence comparable to that in table 1 for this subset of faculty and its Google Scholar citations by gender. Again, Hypothesis 1, that men will receive more citations than women, is supported in 11 of 12 paired comparisons within temporal quartiles. The exception was for the next-to-oldest quartile of women not in top departments whose mean citations were higher than the mean for men, but not significantly so. Again, Hypothesis 1 receives generally strong support.

Table 2 Mean Google Scholar Citations for Political Scientists in PhD-Granting Departments in 2017 by Year-of-PhD Quartiles, Teaching in a Top Department or Not, and Gender^(a)

Notes:

^(a) Cell entries are the mean number of citations from Google Scholar for the subgroup, the N of cases, the range in parentheses, and in brackets a t-test for difference in means when women in a matched male–female pair have a higher mean than men.

^(b) Numbers in the quartiles differ because of tied cases at the cut points.

Source: Kim and Grofman (2019a)

Hypothesis 2, that younger male scholars have especially high citation rates compared to comparable women, also is supported. Again, the youngest age cohort demonstrates especially high gender citation disparities.

Hypothesis 3, that the gender gap will increase over time, again is not supported. There are especially notable—if somewhat uneven—increases in citation parity for “all women” versus “all men” and for those not teaching in top departments. The results in the two columns for those in top departments also demonstrate an overall increase in parity but with a decline from the second to the third temporal cohort pair—perhaps because of a notable male outlier. Yet, over time, women’s citations generally increase to be competitive with if not identical to those of men. Considering citations to more than only journal articles, which notably increases the citations of both men and women, evidently reduces the observed gender disparities.

Surprisingly, the youngest three cohorts of women in top departments are less competitive with their male peers in citations than women in the other column pairs. What accounts for these results is not clear, but they indicate that younger women in top departments experience especially high gender disparity in citations.

Evidence for Hypotheses 4 and 5 with the 2017 data is presented in the online appendix. The percentages of women field faculty are higher in every case than in 2002, yet women remain as only modest subsets of the faculty in every field. Moreover, there is no general relationship between the percentage of women in the fields and gender citation disparities. Those disparities are reduced from those in 2002 for most fields, and women have significantly higher citations than men, on average, in political theory. Thus, citations for women improve when citations to more publications than only journal articles are considered, but women are competitive with men in only one field.

DO OUR CITATION DATA FOR LONG-CAREER SCHOLARS REFLECT SELECTION EFFECTS AGAINST WOMEN?

There is significant concern about the possibility of a “leaking pipeline,” in which more women “abandon the academic pipeline in political science than men” because of dissatisfaction with the academic climate or their professional prospects (American Political Science Association 2005, 3). This possibility could mean that the older cohorts of women in our data might be a select few who have withstood a challenging climate—or enjoyed a good one—and continued to prosper in their career. Thus, they would represent the outcome of a selection effect whereby many less successful women are no longer in the profession or in our data. Good estimates of the causes that could lead to such a situation are rare. However, Kim and Grofman (2019b) investigated career paths from 2002 to 2017 of the scholars in our data for table 1. With information from their article and the online appendix, we constructed and present in table 3 the percentages of men and women in temporal cohorts with at least 100 total cases who remained in PhD departments in 2017, moved to non-PhD departments, or were no longer in academia.

Table 3 Professional Positions in 2017 of Individuals in the Masuoka, Grofman, and Feld Dataset for Faculty in PhD-Granting Institutions at 2002^(a)

Note:

^(a) Excluding those who died after 2002 or were in emeritus positions.

Source: Kim and Grofman (2019b)

The results in table 3 are encouraging with respect to some of the most discussed consequences of a leaking pipeline. The percentages of women who remained in PhD departments, moved to non-PhD departments, and were no longer in academic positions in 2017 are remarkably similar to those for men. Attrition out of the profession is unusually high for young women, but it is comparably high for young men. Kim and Grofman (2019b, 687) observed this as well, based on similar but less comprehensive data. Thus, table 3 suggests that there is no selection effect in these data that could account for our findings. The online appendix explains how these results comport with those of other recent studies of gender disparities in career paths.

…table 3 suggests there is no selection effect in these data that could account for our findings.

CONCLUSIONS

We tested a series of hypotheses for individual scholars’ career citation success with two datasets that have not been used before for this purpose. Our SSCI data include more than 3,000 journals for 1960–2005, and the even more expansive Google Scholar data include publications for 1960–2017. These datasets provide especially comprehensive enumerations of career-long citation records. Using two datasets also provided support for our hypothesis tests by replication.

We found strong support for the hypothesis that, on average, women earn fewer citations than men regardless of their time in the profession and their department of employment. We also replicated the previous finding that younger men have especially more citations than younger women. We also deduced from prior findings on publications and citations that men should continue to outpace women in citations throughout their career. However, the latter expectation is not confirmed. As cohorts of women mature in the profession, they dramatically reduce the citation gender gap in both datasets. What accounts for this remarkable finding should be a major concern for future research.

That young female scholars’ citation rates especially lag behind those of young men and that women make notable over-the-career progress in closing the citation gap may be explained by commonly observed phenomena. Numerous studies propose—and some provide evidence—that young female scholars may not experience the graduate school and early-career mentoring that young men do, enjoy fewer opportunities for coauthoring, submit fewer manuscripts to journal outlets, and shoulder more family responsibilities. Yet, as they mature, many women may overcome these circumstances. Moreover, systematic hypothesis tests for how these circumstances relate to early and long-term citation success may account for several of our findings.

In contrast, our evidence on citation gender gaps in research fields is limited by the modest representation of women in every field. However, these results indicate that some fields provide notably better citations for women’s scholarship, which suggests another avenue for future research.