Political science has no standard for judging what counts as influential research. Yet one thing we do know is that an article that receives no citations is like the proverbial tree falling in the forest: it might as well never have been published in the first place. A 1991 article in Science painted a bleak picture of vast forests of trees toppling without a sound, suggesting that fully 90.1% of all peer-reviewed political science articles received zero citations within five years of publication (Hamilton Reference Hamilton1991). Gary King repeated this figure in his manifesto encouraging political scientists to engage in replication and lamented that only a tiny fraction of all published articles “stimulate active investigation by other researchers” (Reference King1995, 445).
King's lament that the “modal number of citations to articles in political science is zero” has become folk wisdom (see, e.g., Harley et al. Reference Harley, Acord, Earl-Novell, Lawrence and King2010; McGuire Reference McGuire2010), cited as evidence of the poor quality of most peer-reviewed political science articles and the waste of human and financial resources that such lack of quality implies. Fortunately, the folk wisdom is wrong: As a representative of the Institute for Scientific Information (ISI) pointed out in a letter to the editor of Science (Pendlebury Reference Pendlebury1991), the “90.1% are never cited” figure included obituaries, book reviews, notes, editorials, letters to the editor, and assorted other marginalia. When the ISI actually calculates impact scores for its Journal Citation Reports, it only counts citations to research articles and review essays (Garfield Reference Garfield1998).
Even if King's lament is off-base, the question remains: What counts as “impactful” research in political science? Scholars spend countless hours debating journal quality informally; more pertinently, because article publishing is critical for professional advancement and stature, scholars have engaged in efforts to rank journals according to both “qualitative” and “quantitative” impact (e.g., Garand and Giles Reference Garand and Giles2003; Giles and Garand Reference Giles and Garand2007; Hix Reference Hix2004). The qualitative approach asks scholars to subjectively rank average journal quality, while the quantitative approach relies on various permutations of citation counts. Both approaches serve crucial professional purposes: for example, senior faculty use such assessments to debate the fates of junior colleagues, while department chairs and deans use journal rankings to determine salary increases and promotions, as well as to assess whether scholars—and by extension entire departments—are engaged in “high-quality” research or not.
In this article, I seek to change our perception of what “counts” as a citation—and by extension, what counts as a “top” journal—by noting a different, important, and overlooked way that peer-reviewed articles “stimulate active investigation by other researchers.” The rationale behind calculating journal rankings and citation counts rests on two assumptions—first, that journal articles are the main form of academic research output in political science as opposed to books, and second, that counting the number of times an article is cited in other articles is the main method of assessing the impact or quality of political science research. Such an approach implies both that articles with high citation counts are more impactful and that journals with high average citation counts are more influential and intellectually prestigious.
More citations may indeed mean more impact. However, this method of ranking articles and journals discriminates against articles that gain citations in books rather than—or in addition to—citations in other articles. In fact, the average article currently receives approximately the same number of citations in other articles as it does in books. As I discuss in the following, although some citations might “count” more than others, this principle holds true for citations in both articles and books. Regardless, a focus on citation counts from tools such as the Social Science Citation Index (SSCI) ignores a key way that journal articles resonate intellectually and stimulate further inquiry within the discipline. By extension, ignoring citations that articles receive in books also discriminates against those journals that publish articles that tend to gain more attention in books than in other journals. If we rank journals by their average number of citations in books rather than their average citations in articles, their relative rankings change, often quite dramatically: only 11 of the top 20 journals (according to average citations in articles published in 2003) remain in the top 20 when we count average citations in books instead. As I demonstrate in the following, such variation derives from different citation patterns across subfields.
The implication of such shifts is that existing quantitative rankings of journal impact are inadequate. Furthermore, ignoring citations that articles receive in books tends to discriminate against the professional contributions of scholars who publish such articles. Scholars should receive proper credit for the influence of their ideas and hard work; such recognition involves counting the citations that articles receive in published books, as well as their citations in other articles. To the extent that political science is a field in which publication in books and articles matters for one's scholarly reputation, we must supplement indicators of journal impact that are based exclusively on citations in peer-reviewed articles with measures that more fully monitor research performance.
The findings I present in the following will interest deans, department chairs, tenure letter-writers, journal editors, and, of course, every scholar who has ever published an article in a political science journal. Some scholars may be disappointed because some journals' relative disciplinary impact declines when citations in books are taken into account. However, other scholars may be pleased to discover that journals in which they have published—journals that some colleagues disdain as not worthy—leap upward in terms of de facto scholarly influence.
LIMITS OF CONVENTIONAL APPROACHES TO COUNTING CITATIONS
To generate the information in its SSCI and Journal Citation Reports, the ISI counts citations to articles in a list of journals that it considers to have “high impact” within a given discipline. Several scholars have highlighted weaknesses with this approach to assessing articles' impact (e.g., Benoit and Marsh Reference Benoit and Marsh2009). For example, even though its coverage is wide and growing, the ISI's database remains selective, meaning that its SSCI does not count every citation an article might receive in published articles.
The ISI claims that scholarly influence is best measured by only including citations listed in “top quality” journals, but of course what counts as a “top” journal is a matter of debate, and it is not clear how or why the ISI decides to include a journal in its list (Hix Reference Hix2004). For example, Garand and Giles (Reference Garand and Giles2003) surveyed political scientists and compiled a list of 111 journals that respondents regarded as relevant to the discipline but found that ISI only compiled citations for 90 of these (Giles and Garand Reference Giles and Garand2007). Many journals the ISI excludes are certainly obscure, but the point remains that citation counts that use the SSCI will exclude some unknown number of citations to the articles in the ISI's own index. Citation counts based on information from the SSCI are biased toward journals that the ISI—rather than political scientists—defines as impactful.Footnote 1
This article focuses on a different drawback of measuring article impact and/or journal quality: popular citation-count engines do not include citations that appear in books. Rankings of article and journal impact that rely on the SSCI or similar tools, such as Scopus, are based exclusively on the number of citations that an article receives in other peer-reviewed articles. Partly for this reason, the ISI advises an open-minded approach to using its Journal Citation Reports, because “citational practices may vary across disciplines, sub-fields within disciplines, and journals” (Giles and Garand Reference Giles and Garand2007, 741). This is true: political science is a discipline in which books and articles are both valid and important forms of research output. As such, citations both in books and in articles should count as valid and important measures of scholarly impact and influence. Efforts to assess articles' impact that rely exclusively on citation counts in other articles underestimate the total number of citations that an article receives in published research and thus tend to underestimate an article's intellectual impact.
Efforts to rank journals based on article citation counts alone may also prove misleading, because there is no correlation between the number of citations an article receives in other articles and the number it receives in books. Articles are more often quantitative, while books are more often qualitative or multi-method. Journals typically occupy particular intellectual niches and tend to publish certain kinds of articles rather than others. Thus, differences in modes of inquiry provide good reason to suspect that citation patterns will differ across journals and subfields.
In our discipline, the relative frequency with which articles are cited in articles versus books is terra incognita. Yet, when we do not know whether articles resonate in books as well as in other articles, we lack the ability to benchmark the overall “quality” of scholarly research production (Nederhof Reference Nederhof1989; Nederhof Reference Nederhof2006). To the extent that some journal articles are cited more frequently in books and less frequently in articles, any quantitative measure of impact that includes only citations in other articles discriminates against what might be considered highly impactful research and provides a misleading view of scholarly productivity, quality, and impact.
COUNTING CITATIONS OF ARTICLES IN BOOKS
The advent of Google Scholar and Google Books has dramatically changed how scholars conduct research (Kousha and Thelwall Reference Kousha and Thelwall2009). Google Scholar provides search results of academically related materials that are posted to the Web. Its primary advantage is the breadth of its coverage, but this breadth tends to dramatically inflate citation counts by including unpublished material, ephemera such as syllabi and conference programs, and multiple citations to the same article (Jacso Reference Jacso2005; Jacso Reference Jacso2006). A more important problem for present purposes is that Google Scholar dramatically undercounts citations of articles in books.
Consider one example, my article “The Gubernatorial Coattails Effect: Federalism and Congressional Elections in Brazil,” which was published in the Journal of Politics in 2000. According to Google Scholar, in June 2010, this article had 80 citations, including 23 articles (six of which were duplicate citations), five books, and 52 other unpublished items (many of which were also duplicates). However, Google Books found 24 citations to that same article: all five books that Google Scholar found, plus 19 others, with no duplication. In short, Google Scholar does not provide an accurate count of citations, either to articles or to books.Footnote 2
Google Books is a separate and distinct database from Google Scholar. The items in Google Books are not hyperlinked documents posted to the Web; rather, they are scans of physical books contained in actual libraries around the world. To build its books database, Google contracted with several libraries to manually scan millions of books, page by page. To find the number of citations an article has received in books, a user must conduct an “advanced book search” at http://www.books.google.com, entering an article's full title in the “with the exact phrase” box near the top of the page. To count book citations for this study, I used the following parameters: (1) search for “all books,” (2) limit search results to “books” (eliminating “magazines” from the results), and (3) limit results to books in English. I also limited search results by date, as described later.
Google has adopted an all-encompassing approach to its Books project, scanning literally everything on participating libraries' shelves. This approach generates the question of what should be counted as a book. To keep the comparison with articles in SSCI fair, I define a book as something that (1) is not a serial, (2) is formally published by an academic or commercial press, and (3) is not a mere compilation of bibliographic information. Libraries contain many things that are not “books” as I have defined them, but that still contain citations to academic articles, such as think-tank working papers, dissertations, bound copies of journals, and bibliographic reference tools such as Current Contents. For example, I found that nearly all Google Books search results return citations from old issues of journals, hard copies of which remain on library shelves and which Google codes as books rather than magazines. I also discovered that some foreign-language publications slip through the language filter.
To obtain an accurate count of the citations that an article receives in books, one must comb through the Google Books search results by hand and eliminate journals and other “nonbooks.”Footnote 3 After this is done, Google Books provides a count of an article's impact in published books. (I never encountered any duplication of results in Google Books, as is common in Google Scholar.Footnote 4) All methods of counting citations have flaws; as many have noted, the SSCI is also imperfect. Still, to the extent that scholars are interested in discovering the impact their work has in books as well as articles, Google Books opens a window into unexplored terrain.
To compare the number of citations that an article receives in articles and in books, and then to compare journal rankings based on average article and book citations and scholars' subjective assessments, I started with Giles and Garand's (Reference Giles and Garand2007) set of disciplinary journals indexed by the ISI. I then compiled the list of all articles those journals published (if they existed) in the first issue of the year for three years: 1983, 1993, and 2003. For 1983, the SSCI indexed 75 of the 89 journals, resulting in 471 articles. For 1993, 79 of the 89 journals existed, which would have resulted in a list of 494 articles, but the SSCI indexed only 70 of these journals, resulting in 433 articles. In 2003, the SSCI indexed 86 of the 89 journals, resulting in 593 articles.Footnote 5 For each article, I manually compiled the number of SSCI citations to each article after five years. Thus, for articles published in the first issue of 2003, I included citations up through December 31, 2008. I then repeated the exercise using Google Books.
FINDINGS
Let us first explore the broad patterns of citation counts for articles in political science journals. The first piece of information readers will want to know is “how many” citations do political science articles receive in other articles versus the number of citations they receive in books, on average, after five years? Table 1 presents summary statistics for the three years considered.
Table 1 Citations to SSCI-Indexed Articles after Five Years
Including citations that articles receive in books will boost the egos of most article authors, because doing so tends to increase an article's citations by 100% or more. Two things particularly merit note about the information in table 1: first, the mean number of citations that articles receive in other articles has doubled since the 1980s, and second, the number of citations that articles receive in books has always exceeded the number of citations that articles receive in other articles.Footnote 6 It is not clear what accounts for the increase in citations in articles; there are many variables involved, including the fact that the ISI now indexes more journals and has indexed more back issues of journals today than it did 20 years ago.Footnote 7 A key factor in this increase, however, appears to be changes in professional practices within and across political science subfields. To illustrate, I used articles' abstracts to classify them as falling into one of political science's four main subfields.Footnote 8 Table 2 provides the mean number of citations that articles in different subfields received in other articles and in books in each of the three years.
Table 2 Citations in Articles and Books, by Subfield
Breaking down citation patterns by subfield reveals important changes that have occurred over the past 20 to 25 years. The biggest change has occurred in the American politics subfield, which accounts for about 35% of all articles in each year. Articles published in American politics are now cited in other articles almost twice as often as they were 20 to 25 years ago—but those articles are now cited only about half as often in books. Articles have increased in professional importance within this subfield at the same time that books have grown relatively less important. However, this pattern does not hold for the other subfields, in which books continue to be relatively more important sources of citations for published articles (even though citations in articles have also grown in importance in comparative politics and international relations [IR]).
To return to the aggregate picture, although it is not true that 90.1% of all peer-reviewed articles receive zero citations in other articles, it is true that for all three years surveyed, the modal number of citations that articles receive in other SSCI-indexed articles after five years is zero. As the summary statistics imply, a small percentage of all articles brings the average up. Thus, although the mean number of citations has slowly crept upward, most peer-reviewed articles are cited infrequently. Even so, this image of intellectual futility is partly misleading because it ignores the difference that book citations can make for an article's scholarly impact. A focus on citations in other articles fails to consider the possibility that an article's total citation count could be robust, even if it receives zero citations in other articles. Table 3 presents the distribution of the citations that articles received as percentages of all articles for 2003 (the distribution is similar for 1983 and 1993). The first column lists the number of citations that the article received. The other columns list the percentage of articles that received that number of citations (1) in other articles, (2) in books, and (3) by adding citations in articles and books together.
Table 3 Distribution of Citation Counts, 2003 (%)
Given our disciplinary folk wisdom, it merits pointing out that the percentage of articles that receive zero total citations is relatively small. The mean number of total citations in 2003 was 11.14, presenting a somewhat more heartening picture than the impression produced by counting only citations in other articles.Footnote 9 In short, we do article authors a disservice by not counting their work's full intellectual contribution. Indeed, by not including citations in books, article authors fail to account for most of the attention their work often receives.
It is true that this exercise does not take into account potential differences in quality between a citation in an article and a citation in a book, but it is not clear what type of citation should count more. Perhaps a citation in a book should count less than a citation in an article, simply because most books have larger bibliographies than do most articles. However, it is impossible to decide that in the aggregate, one type of citation is more important than the other. Is a citation in a Cambridge or Harvard University Press book more impactful than a citation in the last-ranked journal on the SSCI list? Probably. But is a citation in an obscure journal more impactful than a citation in an equally obscure book? The answer is unclear—after all, scholars in different subfields think that journals in the other subfields are obscure. Even within subfields, scholars differ in their opinions on which journals and books are prominent and which are obscure. Likewise, is getting cited in a Cambridge book more impactful than getting cited in the American Political Science Review? Again, the answer is unclear. The question of whether one type of citation should count more than the other has no answer—but the important point is that both sorts of citations indicate scholarly impact, giving us good reason to count citations in both articles and books.
DIFFERENT METRICS FOR RANKING JOURNALS
How do different articles and journals perform when we apply different metrics to measure their intellectual impact—citations in other articles in the SSCI versus citations in books? There is no necessary correlation between the number of citations that an article receives in articles and the number it receives in books. As a result, journal rankings may change based on whether we rank journals by article citation counts or book citation counts. The appendix ranks all journals in Giles' and Garand (Reference Giles and Garand2007) listing based on mean book citations for 2003. To simplify the exposition, in this section, I focus on how switching from article to book citations changes the rankings of our “top” journals.
Examples set the stage for this comparison. Every year, some articles receive many citations in both articles and books. For example, Fearon and Laitin's “Ethnicity, Insurgency and Civil War” (Reference Fearon and Laitin2003) received 258 citations in other articles, as well as 204 citations in books after five years, placing it at the head of both categories in 2003. If book and article citations were perfectly correlated, we would be satisfied with journal rankings derived from the SSCI or similar tools. However, this correlation does not always hold. Sometimes articles receive more citations in other articles than they do in books. For example, Holbrook et al.'s “Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires” (Reference Holbrook, Green and Krosnick2003) received 49 citations in other articles after five years (nine times the average for that year) but only 11 citations in books (twice the average). Likewise, articles sometimes win more citations in books than in articles. Alastair Iain Johnston's “Is China a Status Quo Power?” (Reference Johnston2003) gained 27 citations in articles after five years (about five times the average) but received 110 citations in books (almost 20 times the average). In short, relying exclusively on citations in articles to assess articles' or journals' relative impact offers an incomplete and potentially misleading view of article and journal impact.
In table 4, I compare the top 10 journals using three different metrics: (1) the mean number of citations that articles in that journal received in other articles from 2003–08, (2) the mean number of citations that articles in that journal received in books over the same period, and (3) political scientists' subjective assessment of that journal's quality.Footnote 10 The first comparison ranks the top 10 journals by mean citations in articles and compares that ranking against another ranking based on mean book citations and political scientists' qualitative assessments. The second comparison takes the top 10 journals according to book citations and compares them against the other two rankings; the third does the same for the qualitative rankings.
Table 4 Comparison of Journal Rankings (N = 85)
Table 4 reveals that journals that publish articles that are cited frequently in other articles do not necessarily publish articles that are cited frequently in books. Moreover, the mean number of citations—whether in articles or books—is not necessarily perfectly correlated with political scientists' assessment of journal quality. This comparison also reveals that some journals climb in the rankings when we use book citations instead of article citations—and some journals fall. To illustrate, table 5 lists the 20 journals from 2003 that decline the most when we shift the ranking from mean article citations to mean book citations.
Table 5 “Decline” in Journal Ranking Using Book Cites rather than Article Cites
The journals that decline the most when using average citations in books tend to publish articles in American politics, quantitative political science, or formal theory. In contrast, journals in the subfields of political theory, comparative politics, and international relations publish articles that tend to gain more citations in books than they do in articles. In particular, articles in area studies journals are more frequently cited in books than in articles. To illustrate, table 6 presents the 20 journals that climbed the most in the rankings when using book citations instead of article citations.
Table 6 “Increase” in Journal Ranking Using Book Cites rather than Article Cites, 2003
In the end, we are left with a confusing picture: most of the top journals in terms of article citations are not the same as the top journals in terms of book citations. Table 7 illustrates that, at most, eight journals in a given year are in the top 20 journals simultaneously in terms of average citations in articles, average citations in books, and political scientists' subjective assessments of journal quality. (These are in no particular order.) Only three journals consistently make the top 20 in all three years: the American Political Science Review, World Politics, and International Studies Quarterly. These publications can be considered the only consistently high-impact political science journals in terms of quantity of citations in both books and articles and perceived article quality. Note that since 1993, with the exception of the American Political Science Review and Public Opinion Quarterly, every journal that has made the top 20 in all three categories publishes in the subfields of comparative politics or international relations.
Table 7 Highest Impact Journals: Journals that Make the Top 20 in All Three Rankings
THE RELATIONSHIP BETWEEN QUALITATIVE AND QUANTITATIVE RANKINGS
The shifts in journal ranking based on article versus book citations that appear in tables 5 and 6 raise the question of how political scientists make subjective assessments of journal quality. In theory, qualitative and quantitative journal rankings should be related; even supposing that some articles are highly cited for infamy rather than quality, it is reasonable to suppose that scholars will tend to esteem journals that publish frequently cited articles.
However, several related questions must be addressed. First, are surveys of political science journal quality subject to a selection effect in which scholars whose careers depend more on article publishing participate at greater rates than scholars whose careers depend less on article publishing? Are participants' rankings of journals also biased for the same reason? We are not likely to find answers to these questions. Yet even if there is no selection effect, do qualitative rankings of journals overlook articles' impact in books? Given our obsession with counting the number of citations that articles receive in other articles, we might be relatively less aware of the impact our work has in books. If this is true, then perhaps participants in surveys that rank journal quality have a tendency to rank certain journals erroneously because of a lack of information about articles' overall impact.
To explore this possibility, I used the Spearman rank correlation statistic to correlate journal rankings based on mean SSCI citations, mean book citations, and scholars' assessments. The results, shown in table 8, suggest that political scientists have a mild tendency to rank journals based on their impact in other journals. In table 8, the key comparison is drawn between the SSCI citation rank and the Google Books citation rank for each year against the qualitative ranking. In each year, the correlation between scholars' subjective rankings and rankings according to SSCI citations is stronger than the correlation between scholars' qualitative ranking and ranking by book citations. The difference was virtually nonexistent in 1983 but was present in 1993 and 2003. When one considers the increase in the relative importance of article publishing in the American politics subfield since that era, the findings in table 8 suggest that rankings of journal quality may be driven increasingly by scholars in American politics. Although these findings provide at best highly indirect evidence, they do suggest that scholars should reconsider journal quality in light of articles' and journals' relative ability to garner citations in books, as well as in articles.
Table 8 Spearman Rank Correlation between Journal Rankings
CONCLUSION
Authors of peer-reviewed articles should receive proper credit for the impact that their work has on future research. Such influence comes in many forms, but despite its limitations, we often rely on the only quantitative measure we have—the citation count. As such, it is in every scholar's interest that every citation “count.” For too long we have relied exclusively on counting citations in articles to assess articles' impact and journals' relative stature, neglecting the citations that articles receive in books. Ignoring articles' influence in books leaves too much at stake: tenure, promotion, job offers, and research funding, for example. We do not neglect books and book chapters when we attempt to quantify scholars' research output; we should not ignore books and book chapters when we measure the impact of such research.
The findings in this article suggest that Google Books offers a valuable new source of citation data for article authors.Footnote 11 All scholars can consult Google Books to determine the number of citations that their work has received when applying for promotion and tenure. Department chairs and deans should take citations in books into account when considering such cases, and journal editors can learn that the articles they publish have influence in previously unknown ways. If one accepts my definition of a “book,” then an idea of an article's relative success can be derived by conducting a Google Book search, deflating the raw result by one-third, and then comparing that figure with the numbers in tables 1 and/or 2.
Measuring scholarly influence by focusing exclusively on citations in articles inflates some articles' overall impact but underestimates the scholarly impact of articles that gain greater attention in books—particularly those in political theory, comparative politics, and international relations. Our discipline encompasses multiple and distinct epistemological and methodological approaches, some of which focus on producing articles and others on producing books. Political science's intellectual diversity requires a broader measure of evaluating the quality of articles and journals.
Unfortunately, Google Books cannot be used for automatic assessment of the relative impact of an article or journal like the SSCI or similar tools, and this article has provided only three “snapshots” of citation patterns. However, the research conducted here could be repeated at relatively low cost at different points in the future to provide evidence of stability or change in article-citation patterns in books. In any case, this article provides a useful empirical baseline for scholars to assess their work's impact in books as well as in articles, and for faculty to use when considering colleagues for promotion and/or tenure.
APPENDIX: Journals Ranked by Average Book Citations, 2003–2008
