Will Open Access Get Me Cited? An Analysis of the Efficacy of Open Access Publishing in Political Science

Abstract

The digital revolution has made it easier for political scientists to share and access high-quality research online. However, many articles are stored in proprietary databases that some institutions cannot afford. High-quality, peer-reviewed, top-tier journal articles that have been made open access (OA) (i.e., freely available online) theoretically should be accessed and cited more easily than articles of similar quality that are available only to paying customers. Research into the efficacy of OA publishing thus far has focused mainly on the natural sciences, and the results have been mixed. Because OA has not been as widely adopted in the social sciences, disciplines such as political science have received little attention in the OA research. In this article, we seek to determine the efficacy of OA in political science. Our primary hypothesis is that OA articles will be cited at higher rates than articles that are toll access (TA), which means available only to paying customers. We test this hypothesis by analyzing the mean citation rates of OA and TA articles from eight top-ranked political science journals. We find that OA publication results in a clear citation advantage in political science publishing.

As academic publishing transitions from print to electronic periodicals, the process by which political scientists share research has become increasingly easier. Considerable high-quality political science research is now available at the click of a button; however, the majority of published articles in the discipline are locked behind paywalls. Subscription rates to journals and scholarly databases are increasingly expensive, putting them out of reach for budget-conscious researchers and institutions. As a result, traditional publishers have developed a number of methods to make their journal articles freely available online; the industry refers to this as open access (OA) publishing. Many proponents of OA argue that this greater accessibility gives OA articles a citation advantage over toll access (TA) articles that are not freely available (Antelman 2004; Davis 2011; Gargouri et al. 2010).

It is important to note that the term “open access” signifies different things. For many, “freely available” scholarship is associated with working papers, non-peer-reviewed work, and online journals with lax peer-review standards. In this article, however, we examine articles that were published in top-tier journals and were converted into OA scholarship because they were self-archived on authors’ personal websites and/or in institutional repositories.¹ This is what the industry refers to as “green” OA (Suber 2012).² In our study, these articles were made “green” (i.e., available to the general public free of charge) after rigorous peer review.³ The eight journals included are American Political Science Review, American Journal of Political Science, Public Opinion Quarterly, Journal of Conflict Resolution, Political Analysis, Political Geography, Annual Review of Political Science, and Comparative Political Science.

Our research question was straightforward: Will making an article OA increase the number of citations it receives? The social sciences have been slow adopters of OA (Calise et al. 2010). In contrast, mathematics, engineering, and the natural sciences were early adopters; therefore, the question has been examined in those disciplines. Results are mixed but generally positive regarding advantageous OA citation effects (OACE) in those disciplines (Doty 2013). However, to the best of our knowledge, political science publications never have been the sole focus of research into the OA citation advantage. This study contributes to the ongoing debate regarding OA efficacy and also serves as a starting point for serious research on OA in political science.

Our primary hypothesis is that OA articles in the discipline will be cited at higher rates than articles that are not freely available online. We test this hypothesis by examining the citation rates of all articles published in the eight journals during a two-year period. We first determined which articles had been made OA and which were still TA. We then examine mean citation rates of the OA and TA articles to determine whether there is a citation advantage for the OA articles. The fact that some journals have permissive self-archiving policies whereas others are more restrictive adds variation to the data, as does the fact that authors may choose not to take advantage of OA policies and therefore will not self-archive their work. Given these parameters, we further hypothesize that the more permissive self-archiving policies of top publishers will result in a greater citation advantage. This article proceeds as follows. First, we articulate our argument in the context of the extant OA literature. This is followed by a discussion of our data and methodological considerations. Then, we present our results and discuss their implications for OA publishing in the discipline. We conclude by noting possibilities for continued research on the efficacy of OA publishing in political science.

OPEN ACCESS AND CITATION ADVANTAGE

Since at least the 1980s, subscription prices of US academic publications have been rising faster than the rate of inflation (Dingley 2005). As a result, academic institutions—particularly college and university libraries—have had to alter purchasing strategies for acquiring and/or renewing databases, journals, monographs (i.e., books), and other academic resources (Greco et al. 2006). This situation, commonly referred to as the “serials crisis,” has had an impact across the spectrum of academic publishing (Greco et al. 2007). In response to the crisis, some called for scholars to reconsider publishing with inflationary and overpriced journals (Parks 2002) and for academic libraries to make access to them as economically efficient as possible (Pascarelli 1990). However, no alternative has shown as much traction as OA publishing.

For the purposes of this study, we use the classic definition of OA, in which an article “is available online to be read for free by anyone, anytime, anywhere—as long as they have Internet access” (Crawford 2011, 1). Despite the broad definition, we have narrowed the OA field by focusing solely on articles published in top political science journals. None of these journals is freely available in toto (which is referred to as “gold” OA). By definition, this restricts our study to green OA, which means articles that have been self-archived by the author and/or a sponsoring institution.

Despite the broad definition, we have narrowed the OA field by focusing solely on articles published in top political science journals. None of these journals is freely available in toto (which is referred to as “gold” OA).

OA advocates argue that disciplines will benefit from free access to information (Calise and De Rosa 2008; Papin–Ramcharan and Dawe 2006) and that the increased use of OA publishing will help to stem the serials crisis (Calise and De Rosa 2008; Papin–Ramcharan and Dawe 2006; Harnad and Brody 2004; May 2005)In addition, advocates argue that OA will increase research efficacy as measured by citation counts and/or the citation impact factor. Despite the budgetary imperative to reduce costs and the potential citation advantage for authors, buy-in for OA publishing has largely been limited to disciplines in the physical and natural sciences as well as engineering and mathematics. Because these disciplines have been heavily invested in OA for many years, they have been the focus of the preponderance of research into the OACE.

Although researchers generally have found a positive correlation between OA and research impact, note that results are varied and the issue is not settled (Doty 2013). For example, Lawrence (2001) found a citation advantage for OA articles in computer sciences, whereas Anderson et al. (2001) found no citation advantage for OA articles in medicine. In subsequent research, Harnad and Brody (2004) reported a citation advantage for OA articles in mathematics and physics, whereas Kurtz et al. (2005) found no advantage in astrophysics. More recent research, however, has provided more support for the conventional wisdom that OA benefits citation counts. For example, a four-discipline study (i.e., economics, applied mathematics, ecology, and sociology) found that the OACE is positive across all four disciplines but that the degree varies among them (Norris et al. 2008). Also, in recent unpublished research, McCabe and Snyder (2014) found a citation advantage for science journals with OA publishing models, although this increase was possibly a “superstar effect” because it was found primarily in the higher-tier journals.⁴

Despite our previous reference to a study that included economics and sociology (Norris et al. 2008), only a few OACE studies have included the social sciences. This is commonly attributed to the low adoption of OA in these disciplines. However, recent scholarship indicates that the prevalence of OA publishing is increasing rapidly in the social sciences generally and political science specifically (Nentwich 2008). Indeed, a recent study (Gargouri et al. 2012, 5) estimates that between 1998 and 2006, approximately 28% of social science articles were OA; that average increased to 36% when the authors studied the data from 2005 to 2010. In addition, the number of OA political science journals listed in the Directory of Open Access Journals increased by more than 50% between 2010 and 2013 (Bjørnshauge et al. 2013; Calise et al. 2010).⁵ Online self-archiving and archiving in institutional repositories complicate the efforts to quantify the volume of political science articles that are OA; the estimates vary from 5% (Hajjem et al. 2005) to 30% (Antelman 2006). Despite this ambiguity, it is generally accepted that between the increasing number of OA journals and the more liberal self-archiving policies that many journals are adopting, the percentage of political science articles that are OA is increasing (Nentwich 2008).

However, as in the natural sciences, the literature presents varied results on the efficacy of OA in different social science disciplines. For example, Antelman (2004, 375–76) found that whereas political science accounted for only 29% of the OA articles, those articles received the highest citation advantage in her study. Similarly, in a 10-discipline study that included political science, Hajjem, Harnad, and Gingras (2005) found that OA articles consistently have more citations than non-OA articles from the same journal and year. However, Evans and Reimer (2009) found that the social sciences receive negligible benefit from OA. Most recently, Xia and Nakanishi (2012) found that anthropologists receive a significant citation advantage from OA publishing independent of journal ranking (i.e., no superstar effect). Although mixed, the greater volume of positive findings in the social sciences leads us to our first hypothesis regarding OA in political science, as follows:

• H₁: OA articles will be cited at higher rates than articles that are TA only.

PUBLISHER SELF-ARCHIVING POLICIES AND OACE

As noted previously, the two primary approaches that academic publishers take to OA publishing are “gold OA” and “green OA.” Gold OA is provided by the journals, often at a cost to researchers or sponsoring institutions. Conversely, green OA is provided by either institutional and/or subject repositories or by individuals posting to their personal or academic websites (Suber 2012, 53). Recently, gold OA was the subject of a symposium at the International Studies Association, the results of which were published in International Studies Perspectives (Gleditsch 2012). In the symposium, Mehlum (2012) makes a strong case for gold OA, citing concerns regarding equality of access for researchers in developing countries.⁶ He also discusses economic and pricing concerns, noting that research could be treated as a public good.

However, a strong counterargument is articulated by both Thompson (2012) and Gleditsch (2012). Although neither author is opposed to OA publishing per se, both note a considerable economic downside for professional organizations and editorial offices. A professional association may receive as much as one third of its operating budget from publishers’ payments for exclusive publication rights to its journal (Thompson 2012). In addition, editorial offices are kept afloat by publisher subsidies and royalties; if association-sponsored journals were to transition to gold OA, Gleditsch (2012) argued, those editorial offices and the quality-control service they provide would be endangered. Thus, gold OA poses serious concerns for associations and their flagship journals.

Within the green OA category, publishers have adopted a wide range of copyright and self-archiving policies.

However, only a fraction of journals—approximately 5% in 2004—would fit the definition of gold OA (Harnad and Brody 2004). Indeed, only an estimated 2% of all OA articles published in 2011 were in gold OA journals; in the social sciences, the number decreased to less than 1% (Gargouri et al. 2012, 6). In contrast, an estimated 90% of journals were “green OA,” giving permission to authors in those journals to self-archive some version of their article (Harnad and Brody 2004). Within the green OA category, publishers have adopted a wide range of copyright and self-archiving policies. Among the larger publishers of political science journals (e.g., Wiley, Oxford, Cambridge, Elsevier, and Sage), the most common policies fall into either the permissive or the restrictive category. For our purposes, “permissive” publisher policies allow authors to self-archive any version of their paper, including the publisher’s PDF. “Restrictive” publisher policies allow authors to self-archive only the preprint (i.e., prior to peer review) version. When publishers explicitly allow self-archiving of the published version of an article, it is reasonable to expect that more authors and institutions will do so. This leads to our secondary hypothesis, as follows:

• H₂: Permissive copyright and self-archiving policies will lead to higher mean citation rates.

DATA AND METHODS

Data and Methodological Considerations

Antelman’s 2004 study of OA efficacy is considered an origin article in OA citation-analysis research, and it provides the earliest evidence of the efficacy of OA publishing. Antelman’s work remains one of only a few articles that specifically address OA and political science. In her 2004 study, political science was studied in contrast with philosophy, engineering, and mathematics. As discussed previously, Antelman found that political science registered the highest effect of OA on mean citation rates. However, her N was relatively small (i.e., 299 political science articles) and her study did not control for journal influence. Along with a lack of fixed time periods and self-selection bias, this is one of the three most-oft-cited methodological problems in the study of OA impact (Craig et al. 2007). We follow Antelman (2004) in testing our hypotheses by comparing mean citation rates; however, we improve on the outstanding methodological challenges.

First, when a researcher does not account for journal influence, articles published in more influential journals are compared to those published in less influential journals. We control for this issue by including only those that were consistently ranked among the top 20 political science journals in the Journal Citation Report (JCR) between 2007 and 2011. This ensures that only journals with comparable levels of influence are included in the study. It is important to note that by including these journals, we make no assumptions about quality; we simply use the JCR ranking as a measure of research impact.

Second, in the absence of a fixed period, a study will include older articles compared to recently published articles; this can skew the citation counts and bias the results. We control for this by limiting the articles to those published only in 2007 and 2008, which ensures that those included in the study have had time to become widely circulated and cited. Another advantage of this approach is that it allows us to compare mean citation rates not only for OA versus TA articles from similarly ranked journals but also for OA and TA articles within the same journal for the same period (Harnad and Brody 2004).

The third major methodological criticism of the extant literature is that when an OA advantage is found, it may be the result of self-selection bias. Critics argue that the “biggest names” are more likely to self-archive and that all authors are more likely to archive their best papers (Hajjem et al. 2005). However, as several other scholars have pointed out, the self-selection–bias argument is flawed (Antelman 2004; Gargouri et al. 2010; Hajjem et al. 2005). First, Antelman dismisses the idea of the biggest names self-selection bias, noting that authors tend either to self-archive all or none of their work. Second, if self-selection of “best papers” were the primary causal factor, it would be logical to expect that self-archived OA articles have higher citation rates than those for which OA is mandatory.⁷ However, Gargouri et al. (2010) tested the citation advantage for mandatory-OA articles versus voluntarily OA articles and found no self-selection bias. The third flaw in the self-selection–bias argument is that researchers must have access to articles to cite them; however, given the serials crisis in academic publishing, no research center can afford to purchase subscriptions to all journals (Hajjem et al. 2005). Thus, it is logical to conclude that OA rather than self-selection is a causal factor in citation-advantage findings.

Data Collection

We began by examining the JCR data to determine which journals were consistently ranked (by impact factor) in the top 20 political science journals between 2007 and 2011. Only eight journals are ranked in the top 20 in all five years (table 1). We use the JCR rankings to select our data sample because impact factor is an indicator of researchers’ use of a specific journal (Antelman 2004). In addition, we used only journals that were highly ranked during a five-year period because the JCR impact-factor score can be unduly influenced by a few highly cited articles (Seglen 1997). As a result, each year’s top 20 rankings included journals that benefited from a high number of citations for a small number of articles rather than a high number of citations for the journal as a whole. Given these issues in the impact-factor calculation, a high impact factor in one JCR report is not a guarantee that a journal is regularly consulted in the discipline. However, high impact factors during a five-year period are a reasonable indicator that political scientists consistently cited articles from the eight journals at high rates.

Table 1 Annual Journal Citation Ranks, by Journal (2007–2011)

After comparing the 2007–2011 JCR reports to determine which journals were top-ranked in all five years, we gathered all of the articles published in them in 2007 and 2008. Then we used the Publish or Perish (PoP) software program (Harzing 2007) to query Google Scholar and retrieve the citation counts for each article in the dataset. We excluded book reviews, letters to the editor, conference programs and proceedings, presidential addresses, and membership meeting notes. We included replies to other authors because they are often cited in literature reviews. In the event that the PoP query returned duplicate citation-count records, the record with the lower count was excluded from the sample. We then used PoP’s built-in Google Scholar queries to determine whether each article in the sample is openly available, in any format, online. These formats, as defined in the Sherpa-RoMEO Publisher Copyright Policies & Self-Archiving Database, include the following:

• • Preprint: the version of the article before peer review

• • Postprint: the version of the article after peer review, with revisions made

• • Publisher’s postprint: the publisher’s PDF version (postpublication)

We then used Sherpa-RoMEO to determine whether the journal is subject to permissive or restrictive self-archiving policies.⁸ The end result was a database of 727 observations, each representing a single article and its citation count.

Descriptive Statistics

Our results must be understood in the context of the data sample; therefore, it is important to include descriptive statistics. As noted previously, our sample size was 727 articles. Of those, 404 articles (55.5%) were OA in some form. This is surprising given that much of the extant literature indicates that the social sciences have not adopted OA publishing at high rates. Although it is true that no mainstream political science journals have converted to gold OA (Bjørnshauge et al. 2013), the data presented here indicate that individual political scientists are making publications OA at fairly high rates. Overall OA frequency is reported in table 2a.

Table 2a Open Access Frequency

Note: N = 727. Percentages may not total 100% due to rounding.

It is remarkable that whereas the majority (more than 75%) of OA articles in the sample were publisher PDFs, more than 45% of them were from restrictive publishers. This may indicate that authors are either ignorant of or indifferent to the provisions of their publisher copyright agreements.

It is remarkable that whereas the majority (more than 75%) of OA articles in the sample were publisher PDFs, more than 45% of them were from restrictive publishers. This may indicate that authors are either ignorant of or indifferent to the provisions of their publisher copyright agreements. However, it also may indicate that because citations of unpublished work are uncommon in political science, authors self-archive the publisher PDF in the hope of more citations. Furthermore, of the 410 articles published by permissive publishers, 48% were not self-archived. This indicates that political scientists are not taking full advantage of author-friendly copyright agreements. Table 2b shows OA frequency by publisher-policy classification. Table 2c presents the variation in OA frequency and publisher-policy classification by journal.

Table 2b Open Access Frequency Rates and Publisher Policies

Notes: N = 727. Percentages may not total 100% due to rounding.

**P = Permissive; R = Restrictive.

Table 2c Open Access Frequency Rates and Publisher Policies by Journal

Note: *Percentages rounded to the nearest whole number. ** P = Permissive; R = Restrictive.

The descriptive statistics provide prima facie evidence that OA articles are cited more frequently than non-OA articles (figure 1 and table 3). As shown in table 3, the mean citation rate in the sample size is 51. For OA articles, the mean citation rate is 70 and for TA articles, it is 28; thus, the number of citations is approximately two and a half times higher for OA articles in this sample. As shown in table 3, the mean citation rates for OA articles are higher not only across the full sample but also within samples for each journal. The table also shows wide variation in the mean citation rates between journals, from an average difference of only 18 citations between OA and TA articles in Public Opinion Quarterly to an average difference of more than 100 citations in Annual Review of Political Science.

Figure 1 Citation Rates by Open Access Status

Table 3 Comparison of Mean Citation Rates between Open Access and Toll Access Articles

RESULTS AND DISCUSSION

Because our sample cannot be assumed, a priori, to be normally distributed, we used the nonparametric Wilcoxon-Mann-Whitney (WMW) test (also known as the Mann-Whitney U) to test the difference of means (Großer and Schram 2006; Munck and Snyder 2007). We first tested our primary hypothesis: OA will lead to more citations than TA. The WMW results indicate that OA articles have significantly higher mean citation rates than TA articles. This holds across the data sample as well as within each of the included journal; therefore, OA publication results in a clear and significant citation advantage. The results of these tests are presented in table 4. To confirm these findings, we logged the citation rate and conducted an independent samples T-test; those results also indicate that OA articles are cited at a significantly lower rate than TA articles–which again demonstrates an OA citation advantage (T = 11.5, p < 0.0000). Given the citation advantage, these results indicate that if political scientists want to be cited at higher rates, they should publish in journals that freely allow authors to self-archive or advocate for more restrictive journals to offer self-archiving.⁹ Furthermore, our data indicate that almost half of the authors whose copyright agreements allow them to self-archive the final version (i.e., publisher PDF) do not do so. The citation advantage provided by OA publishing indicates that authors should take advantage of the permissiveness of their copyright agreements if they want to be cited at higher rates.

Table 4 Wilcoxon-Mann-Whitney Difference of Means Results, by Data Source

Note: *** p < 0.01, ** p < 0.05.

Next, we discuss our secondary hypothesis: More permissive publisher self-archiving policies will lead to higher citation counts. The WMW analysis, conducted on the full sample and presented in table 5, suggests that permissive publisher policies are associated with higher citation rates. However, further data analysis contradicts this initial result. For example, the journal with the highest OA frequency (Political Analysis, 74%) is subject to restrictive policies, whereas the journal with the lowest OA frequency (Political Geography, 23%) is subject to permissive policies. This suggests that, at least superficially, publisher policies are not a causal factor in citation advantage.

This descriptive finding is confirmed by additional difference-of-means testing. We selected the OA population of 404 records and tested the mean citation rates (by self-archiving permissiveness) within that subset of the sample. The results, presented in table 5, indicate that although positive, there is no statistically significant citation advantage to publishing with a more permissive journal. This is counterintuitive, and may be an artifact of the data. As noted previously, a large percentage (45%) of the OA articles in our sample have been posted in violation of the publisher’s copyright and self-archiving policies, and a similarly large percentage (48%) of the articles subject to permissive policies were not made OA. It is equally likely, however, that lack of education regarding self-archiving leads authors to simply sign the copyright agreement.

The results indicate that although positive, there is no statistically significant citation advantage to publishing with a more permissive journal. This is counterintuitive; this finding may be an artifact of the data.

Table 5 Wilcoxon-Mann-Whitney Difference of Means Results, by Publisher Policy

Note: *** p < 0.01, ** p < 0.05

CONCLUSION

By using standard OACE research methods to evaluate citation rates for the top political science journals, this article marks an important step for the discipline. Our results have implications not only for authors but also for institutions and publishers. As the serials crisis persists, alternative publication models such as gold and green OA will continue to be developed, explored, and evaluated. For authors, wider acceptance or rejection of OA publishing models will hinge on the perceived and measured impact of those publications in their respective academic fields. This study provides evidence that OA is beneficial to political scientists: when researchers find the full-text version of a high-quality article without being prompted for payment, they are more likely to use it in their own research. However, the data indicate that although political scientists seem to be self-archiving at relatively high rates, many are not—even when permissive self-archiving policies allow it. Although our research indicates that publisher permissiveness does not result in a citation advantage, it does indicate an advantage for freely available articles. Thus, political scientists should be self-archiving whenever their copyright agreements permit.

This finding brings up two issues, however. First, researchers often are ill-informed about the details of their copyright agreements. Second, many researchers lack the technical skills or resources that would enable them to self-archive their work. These problems can be solved at the institutional level. Many institutions hire specialists to help authors with copyright agreements and have established repositories to make it easier for authors to self-archive. The benefit to an institution is that self-archiving can raise the research profiles of both the researcher and the institution.

Finally, the results presented in this article also have implications for publishers. As noted previously, we found that articles from more restrictive journals often are self-archived at higher rates than those subject to more permissive policies. This indicates that some political scientists may be self-archiving in violation of their copyright agreements, whether or not knowingly. Although publishers historically have ignored these violations, recent news indicates that some may be less likely to overlook them in the future. In late 2013, science-publishing giant Elsevier served takedown notices to dozens of institutions and websites, demanding that these entities remove articles that had been posted without Elsevier’s permission (Peterson 2013). If more publishers follow Elsevier’s lead, it may become more important that political scientists publish with journals that have permissive policies or push more restrictive publishers for self-archiving exceptions in their copyright agreements. Indeed, if publishers start enforcing copyright agreements more stringently, authors increasingly may favor OA to find a wider audience for their work.

The clear OA citation advantage found in this study indicates that political scientists can increase access to and use of their research by self-archiving. However, more research is needed. Although our dataset includes the largest sample of political science OA and TA articles tested to date, it remains a narrow sample. First, we restricted the time to two years of data to ensure that the observations were comparable in terms of opportunity to accrue citations. Second, there is a slightly wider spectrum of publisher policies than is represented in this study. The dataset includes only those journals with highly permissive publishing policies (i.e., authors can self-archive any version) and those with moderately restrictive policies (i.e., authors can self-archive preprints only). Thus, the study excludes publisher policies that prohibit all self-archiving. While in theory these should be cited at a much lower rate, our study indicates that this is not the likely case; the inclusion of observations from fully restrictive publications would be an interesting expansion of the study. Also, given the superstar effect found in recent research, it would be useful to investigate relative journal ranking as a factor in OA citation advantage. Finally, as the major associations in the discipline enforce their requirement that authors upload conference papers, it will be important to study the effects of this type of preprint on citation rates.