Are Papers Written by Women Authors Cited Less Frequently?

Abstract

Dion, Sumner, and Mitchell (2018) find that a published article is more likely to cite at least one female-authored paper if that article is itself authored by women. To complement their work, we study the number of times that an article in their data set is cited given that it has at least one female author. We find that articles with at least one female author are cited no more or less often than male-authored articles once we control for the publishing journal and the number of authors. The importance of controlling for author count in our model suggests that spurious correlation and/or self-citation might explain at least some of the gender differences found by Dion, Sumner, and Mitchell (2018).

Dion, Sumner, and Mitchell (2018) demonstrate that a published article is more likely to cite a female-authored paper if that article is itself authored by women. ¹ They also show that articles in Political Analysis are substantially less likely to cite female-authored papers when compared to articles in the discipline’s pre-eminent general interest journal, American Political Science Review. ² This difference is not peculiar to political science as a discipline: articles in Econometrica are also less likely than articles in American Political Science Review to cite a female-authored paper. ³

Some of the potential explanations for these results are extremely concerning from the perspective of achieving gender equality in the political methodology community. For example, it may be that women are not as well-networked in the profession as male counterparts, and thus have greater difficulty gaining attention for their work. Or, perhaps authors simply take the work of women authors less seriously. These possibilities represent what Dion, Sumner, and Mitchell (2018, p. 3) call the “Matilda effect.” Either possibility is worrisome.

To complement the findings of Dion, Sumner, and Mitchell (2018), we study the difference in citation rates between male- and female-authored papers. That is, we are interested in the following:

1. a. $(\text{citation count}|\text{at least one female author})-(\text{citation count}|\text{only male author(s)})$ , and

2. b. $\Pr (\text{paper cited}|\text{at least one female author})-\Pr (\text{paper cited}|\text{only male author(s)})$ .

By contrast, Dion, Sumner, and Mitchell (2018) study the probability that an article’s reference list contains at least one paper authored only by women as a function of the gender composition of the citing article’s authors. ⁴ We argue that our dependent variables speak more directly to diversity in the methods community because citation counts are an increasingly important metric for assessing scholarly impact, for example, when evaluating a tenure case (Fowler and Aksnes 2007; Fischer, Ritchie, and Hanspach 2012; Cameron, White, and Gray 2016; Marini 2017). If women systematically receive fewer citations to their papers than men, this could directly harm these women’s career prospects and pose a barrier to diversifying the methods field. For the same reason, we choose to code our key independent variable (“female author”) such that both women-only author teams and mixed-gender author teams are classified as 1, while male-only author teams are classified as 0. Our choice focuses on citations to all female authors, not just those who write alone or with other women, and increases the proportion of “treated” cases in the data set (see Table 1 in Dion, Sumner, and Mitchell (2018)).

We collected citation data from Crossref (www.crossref.org) for all the publications in the Dion, Sumner, and Mitchell (2018) data set using the rcrossref package of Chamberlain et al. (2017). ⁵ The resulting citation count is our dependent variable. We were able to collect citation data for 1907 of the 1939 publications in the data set. ⁶ Papers in this data set had been cited between 0 and 984 times, with a mean of 28.92 citations per paper. As reported by Dion, Sumner, and Mitchell (2018), the data set contains all articles published between the years 2007 and 2016 in five prominent journals, including three methodological journals (Political Analysis, Econometrica, and Sociological Methods and Research), a general interest political science journal (American Political Science Review), and a political science journal focusing on gender (Politics and Gender).

A regression analysis of the citation data is shown in Table 1. Models 1 and 2 use ordinary least squares (OLS) regression to predict the number of citations using whether at least one author is a female, the publishing journal, and the number of authors (model 2 only). ⁷ Model 3 uses OLS to predict whether the paper has been cited more than five times (1 if so, 0 if not); ⁸ this model tries to directly assess $\Pr (\text{paper cited})$ using five citations as a cutoff for “not being cited.”

Table 1. Regression analysis of citation counts for published papers in the Dion, Sumner, and Mitchell (2018) data set.

Results are coefficients from OLS regression (standard errors in parentheses). All models include year fixed effects (coefficients omitted from table). Constant is not included in any model. APSR, American Political Science Review; SMR, Sociological Methods and Research.

The combined evidence of Table 1 indicates that female authors are not at a systematic disadvantage when it comes to being cited, holding the publishing journal and the number of authors constant. Model 1 in Table 1 seems to indicate that articles with female authors receive about five more citations on average than those with only male authors, ceteris paribus. This relationship is sensible: for nearly any fixed probability that a woman is a paper author, a greater number of co-authors implies a higher probability that at least one author is female. ⁹ At the same time, previous research has found that the number of co-authors on a paper is associated with a greater number of citations to that paper—a pattern that holds across multiple fields (Wuchty, Jones, and Uzzi 2007; Franceschet and Costantini 2010; Abramo and D’Angelo 2015). This increase in citations may be due in part to the fact that as the number of authors increases, so does the network of scholars who might know and cite those authors (Leimu and Koricheva 2005, p. 30). We thus have reason to suspect spurious correlation between our “female author” variable and the number of citations; the potential back-door pathway leads through the number of authors of the paper. Once author count is included as a variable in the regression (in Model 2), this relationship becomes statistically indistinguishable from zero and substantively small. The findings of Model 3 (using a binary dependent variable for citations) are substantively similar to those of Model 2. Our overall conclusion is that there is no apparent relationship between author gender and citation count in this data set once author count is held constant.

The importance of the author count variable suggests that confounding might explain at least some of the gender discrepancies found in Dion, Sumner, and Mitchell (2018). Women are underrepresented as solo authors in general (West et al. 2013), tend to collaborate more on average than men (Kumar and Ratnavelu 2016), and tend to have a greater number of co-authors (Bozeman and Gaughan 2011). ¹⁰ However, in the Dion, Sumner, and Mitchell (2018) data set, articles with only women as authors have on average 0.8 fewer authors (with a standard error 0.05) compared to articles with male authors or mixed-gender author teams. Again, this is sensible: for nearly any fixed probability that an author is female, a greater number of co-authors implies a lower probability of an all-female author team. ¹¹ Moreover, it is plausible that there is a positive association between the number of authors of an article and the number of papers cited by that article (Corbyn 2010; Davis 2010); citing more papers presumably means a greater chance of citing at least one paper authored by women. Consequently, there is a possible back-door causal pathway between the probability that all authors on a published article are female and the probability that the article cites another paper with only women authors.

It is also possible that papers with female authors cite women more often because all authors (including women) often cite themselves (Fowler and Aksnes 2007). Even if women cite themselves less often than men (King et al. 2017), a positive propensity for authors to self-cite could explain why a paper with only female authors would be more likely to cite another paper with only female authors.

Although we find that female-authored papers are not any more or less likely to be cited on average once we control for the journal and number of authors, our results do not imply that there is no gender representation problem in the political methodology community. Both Political Analysis and Econometrica have a substantially lower proportion of articles with at least one female author than Sociological Research and Methods, with the latter journal having over 9 percentage points more articles with a female author compared to the former two in this data set. Given the very high citation rates of Political Analysis and Econometrica, both of which are broadly comparable ¹² to American Political Science Review in impact factor, our results suggest that boosting the representation of women in citations could be achieved by increasing the diversity of the methods community through active recruitment and mentoring of, and co-authorship with, people from underrepresented groups (including women) who have an interest in methodology.