Citation patterns in economics and beyond

Argument

In this paper we comparatively explore three claims concerning the disciplinary character of economics by means of citation analysis. The three claims under study are: (1) economics exhibits strong forms of institutional stratification and, as a byproduct, a rather pronounced internal hierarchy; (2) economists strongly conform to institutional incentives; and (3) modern mainstream economics is a largely self-referential intellectual project mostly inaccessible to disciplinary or paradigmatic outsiders. The validity of these claims is assessed by means of an interdisciplinary comparison of citation patterns aiming to identify peculiar characteristics of economic discourse. In doing so, we emphasize that citation data can always be interpreted in different ways, thereby focusing on the contrast between a “cognitive” and an “evaluative” approach towards citation data.

1. Introduction

Scientometrics is the attempt to better understand the nature of scientific discourse by employing and analyzing quantitative data as it emerges from scientific exchanges (Mingers & Leydesdorff 2015). Analysis of citation data has achieved an especially prominent role, as it can be used not only to describe characteristic patterns within scientific conversations, but also serves as a means for assessing the impact and quality of contributions, outlets, authors, or departments.

In this paper we take different ways of utilizing citation data as starting points to better understand the idiosyncratic character of the discipline of economics: more precisely, we compare citation patterns in economics to those of other academic disciplines and ask how different conceptual understandings of citation data also lead to differing interpretations with regard to the nature and comparative development of academic disciplines. Thereby we make use of two basic, but different, ways of interpreting citation data: on the one hand, we follow what is a standard routine in many academic contexts, namely to interpret citations as an indicator for the quality of the cited publication. We call this the evaluative use of citation data (see Moed et al. 2012, and, for specific examples in economics, Arrow et al. 2011, and Diamond 1989). The second approach we suggest is to interpret the act of mentioning one scientific publication in another as an indicator for some form of “communication” between researchers: the act of citing a paper is understood as a key element in scientific conversation. In such an approach, citation analysis allows for assessing the properties of scientific conversation, which amounts to answering questions like “who talks to whom,” “what are the intellectual foundations of a certain field of research,” or “how do scientists distribute their attention across the whole universe of scientific papers?” Such uses of citation data have traditionally been labeled as cognitive scientometrics (De Solla Price 1965; Rip & Courtial 1984; and, for examples in economics, Hamermesh 2018 and Kim et al. 2006), which serves as a second and alternative frame of interpretation applied to our quantitative findings. While these two approaches are not necessarily contradictory – as citations often serve as both, a means for communication as well as an indicator of (dis)agreement, – they obviously emphasize different aspects of the scholarly process and, hence, can lead to quite different or even conflicting conclusions when applied to actual data.

In what follows, we evaluate and compare citation data from five academic disciplines and analyze the respective citation patterns. We are especially interested in differences between citation patterns in economics and those in other disciplines, since economics is typically understood as a highly idiosyncratic field of research. There are several reasons for this peculiar character of economics: one such reason is that economists often consider their field as most advanced and, hence, superior compared to the other social sciences (e.g. Fourcade et al. 2015; Lazear 2000). This attitude is at least partly due to the origin-story of modern economics, which is influenced by the archetype of the natural sciences – a fact that had repercussions on both the historical positioning of the discipline (Samuelson 1972) as well as the evolution of its prevailing practices (Mirowski 1989). Other aspects of the peculiarities of economics include differences in research style and method (owing to a focus on mathematical and statistical methods [Morgan 1988]); a stronger degree of internal stratification (Fourcade et al. 2015; Hodgson & Rothman 1999); its prominent role in the context of political decision-making (Beker 2010, Hall 1989); or its focus on a single, major theoretical approach as summarized in highly standardized textbooks (Graupe 2016; Skousen 1997).

In what follows, we take three of these alleged peculiarities of economics as a starting point for understanding interdisciplinary differences in citation patterns. Specifically, we relate to findings (a) that modern mainstream economics shows greater signs of stratification than other fields of research (Bayer & Rouse 2016; Fourcade et al. 2015; Han 2003; Hodgson & Rothman 1999), (b) that economists are more prone to conform to institutional incentives due to their conceptual focus on instrumental rationality (Bauman & Rose 2011; Etzioni 2015; Frank et al. 1993; 1996; Frey et al. 1993; Yezer et al. 1996), and (c) that mainstream economic research is autonomous in the sense that it only rarely relies on external input (Fourcade et al. 2015; Pieters & Baumgartner 2002) and, hence, is largely inaccessible to paradigmatic or disciplinary outsiders (Dobusch & Kapeller 2009; Fourcade et al. 2015; King 2003; Leijonhufvud 1973). We aim to provide an empirical evaluation of these claims on the peculiarity of economics by comparing citation patterns found in economics with those found in some companion social sciences (sociology, political science, and psychology) as well as in different branches of physics. The main rationale for selecting these fields is that they either cover related subjects and questions (the social sciences) or serve as a historically important conceptual archetype for conducting economic research and composing economic theory (physics). In doing so, we hope to shed some light on the question of whether and to what extent these peculiarities of economics are brought forth by a specific disciplinary culture as opposed to being driven by differences in research effort and quality.

Additionally we argue that economics is of special interest when it comes to questions of institutional power-relations, intellectual stratification, or the distribution of attention within a discipline for at least two reasons: first, we find not only that the discipline of economics is coined by stark internal differentiations and a rather rigid internal hierarchy, but also that these properties affect several dimensions. Examples for this specificity include the popularity and gate-keeping power of specific outlets such as the “Big five” (see Heckman & Moktan 2018), the recruitment of officers and editors in the discipline’s most important institutions (Fourcade et al. 2015), the high degree of institutional concentration in economic publishing (Aigner & Glötzl, in this issue) or the influence of PhD-advisers or co-authors from well-respected institutions on publishing prospects (Colussi 2018; D’Ippoliti 2018).

Second, we argue that the question of how attention, citations, or resources are distributed is especially delicate in economics, as economics is typically understood as a paradigmatically and politically contested field of research (Lee & Elsner 2011). This view in turn raises the question of whether the alleged “contestedness” of economics has an impact on its discursive properties, that is, on the way it organizes and reproduces scientific conversations. It is this strong degree of internal stratification in conjunction with the character of economics as a “contested discipline” that motivated our main hypotheses on the peculiarity of economics and the economic discourse.

In what follows, we analyze three main aspects of current economic discourse – disciplinary stratification, compliance with incentives, and conceptual isolation – by means of an interdisciplinary comparison. Thereby, we expect our results to reflect the disciplinary character of economics: we assume to find higher degrees of discursive concentration, a higher responsiveness towards incentives, and a greater propensity to engage in inward-looking or exclusionary styles of research and discourse. However, while such findings would lend additional support to the diagnosis that economics is an exceptionally insular and hierarchical research field (Fourcade et al. 2015), we also hope to provide some clues on which factors – e.g. disciplinary culture, core theoretical ideas, or paradigmatic prejudices – do contribute to the emergence of these peculiarities of economics as an academic discipline.¹

2. Concentration

In this section we subsume the comparatively high level of self-reference and the strongly hierarchical character of economics under the heading of concentration to signify that important authors, contributions, departments, and outlets are (more) strongly concentrated in terms of their spatial distribution and their personal as well as professional relations. We argue that one of the main roots of concentration in economics is the highly stratified tertiary education system comprised of a very small number of high-profile schools in economics, which in turn have a tight grip on the most important outlets and professional associations.² The hierarchical structure within the economic discipline has been confirmed empirically in various ways. Medoff, for instance, finds evidence of a “positive institutional Matthew effect” (Medoff 2006, 485) for Harvard University and the University of Chicago leading to a disproportionate amount of peer recognition devoted to contributions by economists affiliated with these institutions. On a more general level, Baghestanian and Popov find a strong association between the reputation of an economists PhD-granting institution and the probability of publishing in a top economic journal³ which they call the “Alma mater”-effect.

The Alma mater effect is large in size: a top-ten graduate has a 30 percent chance of publishing in a top-five journal, but if instead he graduates from a top-thirty, his chances decrease to 17 percent, and lowering his Alma Mater ranking further to 100+ lowers his chances to ten percent. (Baghestanian & Popov 2017, 17)

Another finding on “concentration” in economics is due to Hodgson and Rothman who analyze the institutional background and affiliation of the editors and authors of a set of thirty “top” economic journals. They identify a strong dominance of institutions located in the U.S., and, within that subset, a dominance of a small group of elite universities. For instance, in 1995 more than half (54 percent) of the authors of articles published in these “top thirty” journals obtained their PhD degrees at one of twelve prestigious U.S. universities,⁴ while at the same time more than one fifth (21 percent) of authors in said journals were also affiliated with one of these universities (Hodgson & Rothman 1999).

A partial replication of the analysis of Hodgson & Rothmann (1999) with data from 2014 suggests that this level of “institutional oligopoly” has remained fairly constant over time: Just looking at the subsample consisting of three major journals, the Quarterly Journal of Economics (QJE), the American Economic Review (AER)⁵ and the Journal of Political Economy (JPE) reveals that a majority of authors (55 percent) publishing in these outlets received their PhD degrees at one of the twelve universities mentioned above. In 2014, more than one third (38 percent) of these authors were affiliated with one of these universities at the time of publication (see also Wu 2007).

Similar tendencies of concentration can also be found in large-scale analyses of the economic literature. Laband (2013), for instance, analyzes the representation of journals in the reference sections of the 409 most cited economics papers published between 2001-2005: more than half (55 percent) of the references cited by these top-papers came from a group of only twenty journals. Even more strikingly, the proportion of citations devoted to the three journals mentioned above – QJE, JPE and AER – is about one fifth (20.6 percent) of the total references (cf. Laband 2013, F250-F251). In light of these observations, it comes as no surprise that a recent analysis of the development of economic “top-journals,” which has been published in the Journal of Economic Literature, was limited to the study of only five journals, adding Econometrica and the Review of Economic Studies to the set of three journals mentioned above (see Card & DellaVigna 2013).

Our empirical application takes a closer look at the “Big 5,” i.e. the five core journals covered by Card & DellaVigna (2013), to gain potential insights on tendencies of concentration within economic discourse similar to those already observed in the context of editor- and authorships in top-journals (Hodgson & Rothman 1999) or the recruitment of officers for academic associations (Fourcade et al. 2015). Specifically, we measure the concentration of attention within top-journals by assessing the degree of self-referentiality within this group of journals to compare the relative openness of top-journals across fields. Figure 1 plots the share of journal self-citations as well as the share of citation flows originating from within the “Big 5” relative to total citations in these journals for the period 2009-2013. Figure 1 conveys our first finding: more than a quarter of all sources cited by the top five economic journals originate are self-referential – they stem from the very same set of journals.

Figure 1. The intra-group citation behavior of five core economic journals. (The authors’ own calculation based on data from Clarivate Analytics.).

In addition to this key observation – that one out of four citations made in this journal sample also originates from within this sample – it can also be inferred from Figure 1, that the average share of self-citations within a top-journal plus the share of citations imported from its four “best buddies” is fairly stable across the top five journals with individual values ranging from 23.8 percent (QJE) to 31.7 percent (EMA).

When comparing the citation pattern found across economics’ top journals to the degree of self-reference in top journals located in other fields – e.g. the “top five” journals in psychology, sociology, political science, and physics (fig. 2) – there is the difficulty of selecting appropriate sets of top journals for these fields to advance a sensible comparison across disciplines. To enhance comparability of the respective “top” samples in each discipline we first harmonized our selection criteria across all fields under study. Specifically, we selected every journal that was ranked in the top 20 of Clarivate Analytics’ ⁶ annually published Journal Citation Reports (JCR) for ten consecutive years (2006-2015) to account for the relative long-term position of outlets – a condition inspired by the ranking patterns in economics, where the “Big 5” are constantly within the Top 20 of journals as ranked by the JCR. In the case that more than five journals fit those criteria, the five journals with the highest average rank were selected.

Figure 2. An interdisciplinary comparison on concentration – the intra-group citation behavior of five core journals in (a) psychology (multidisciplinary), (b) sociology, (c) political science, and (d) physics (multidisciplinary). (The authors’ own calculation based on data from Clarivate Analytics.).

In comparison with fig. 1, the results shown in fig. 2 reveal two remarkable findings. Firstly, in the other social science disciplines considered here, the degree of concentration measured as the share of journal self-citations plus the citation flow within top journals is – with average values ranging between 7.9 percent and 12.4 percent of total citations – pending between a third and a half of what we found for economics (27.5 percent). This finding reinforces the diagnosis that economics exhibits a much more pronounced hierarchy, or “pecking-order” (Leijonhufvud 1973, 328), compared to other social science disciplines. Secondly, this relative high degree of concentration in economics also exceeds the degree of concentration measured within our sample of (multidisciplinary) physics journals (20.3 percent). Hence, although the difference between economics and physics as its conceptual archetype is less pronounced than the one between economics and the other social sciences, there remains a substantial gap even in comparison to the much different and often more narrow discourses in the realm of the natural sciences.

In response to the finding on concentration in physics documented in fig. 2, we acquired data for analyzing a second example in the realm of physics, but, this time, focusing on top journals in very specific research areas to assess the intuition that economics is similar to specialized, instead of general, discourses in physics. In doing so, we found a striking similarity between our novel sample from physics consisting of specifically chosen field journals (Physical Review A-E), see fig. 3, and economics. Although the network structures differ markedly between these two cases – the Physical Reviews are huge journals with an average publication output of more than 3,000 papers per year focusing on very different subfields, the degree of concentration is nearly as high (27.3 percent) as in the economics sample. While the stronger content-based separation of the respective journals into subfields (designated with letters A-E⁷) provides an explanation for a much higher share of self-citations, we also find that the average amount of self-citations within these journals (24.1 percent) is eventually rather close to the average share of attention the “Big 5” in economics devote to themselves (27.5 percent). In sum, the top five journals in economics taken together behave rather similarly to specific field journals within physics in terms of discursive concentration.

Figure 3. Citation patterns in specialized physics journals – the intra-group citation behavior of five core field journals in physics. (The authors’ own calculation is based on data from Clarivate Analytics.).

The result that the “Big 5” in economics behave similarly to more specialized subfields in physics and exhibit correspondingly high rates of concentration is especially striking in the face of an overall decrease in the concentration of cited sources within the academic literature over the past decades (Larivière et al. 2009). Beside this anachronism, it is quite remarkable that a multi-faceted discipline such as economics, which deals with mathematical, game-theoretical, sociological, historical, philosophical, and psychological components, behaves in the same concentrated manner as, say, Physical Review B, which focuses on condensed matter and material physics. The fact that the relative citation patterns of top multidisciplinary journals in physics (as documented in the lower right panel of fig. 2) shows much less regularity, gives an indication for the tight conceptual and theoretical commitments established in contemporary economics.

While these findings are interesting in themselves, it is worthwhile to further explore how their actual implications depend on the respective frame of reference: in terms of an evaluative interpretation, for instance, the relevant problem is to explain the comparatively strong agglomeration of high-quality articles in such a small number of outlets. A potential conclusion to be drawn from this perspective is that the editors of these journals do excellent work and manage to concentrate most high-quality research in economics in their outlets. This constellation, in turn, makes it easier to locate said contributions.

In contrast, while a cognitive interpretation could take such reasoning into account, it would eventually be forced to ask a broader set of questions and inspect other variables, like the diversity of authors, readers and institutional backgrounds, the role of reputational concerns, or the impact of “Big 5” publications on tenure decisions and related career concerns (Heckman & Moktan 2018). In comparison with physics (fig. 3), such a cognitive perspective would necessarily go beyond asserting that editors in economics are simply superior, but would rather ask which properties make economics discursively similar to subfields in modern physics.

Under such premises, the outcome looks quite different: here the findings above reveal that the economic discourse at the very top is to a large extent self-contained and exhibits a strong hierarchical character, while the discourse in sociology and political science can be characterized as more diverse and, correspondingly, less hierarchical and insular. This example illustrates how the interpretation and associated valuation of citations is guided by theoretical presumptions, which effectively decide how to translate numerical into verbal assessments.

3. Reactivity

The second aspect in this paper relates to the phenomenon of reactivity, specifically in connection with scientific research evaluation. Our basic argument is that the introduction of journal rankings in economics in particular and in academia in general has led to a change in and a refinement of the institutional incentives governing the academic sphere.

Notwithstanding the fact that rankings have a long tradition in economics (see, for instance, Diamond 1989; Fusfeld 1956; Graves et al. 1982; Liebowitz & Palmer 1984; or the rankings hosted by RePEc on repec.org), we argue that the importance of journal rankings received a significant boost within the last two decades (see also Kalaitzidakis et al. 2011; Stern 2013; Zimmermann 2013). The journal rankings originally produced by the Institute for Scientific Information (ISI), whose database is today widely known as Clarivate Analytics’ Web of Science, proved to be pivotal for the ascension of journal rankings in academic assessment and evaluation, which eventually led to visible changes in the disciplinary trajectory of economics at the national level (Lee 2007; Bloch 2010; Corsi et al. 2010). As these rankings are imposed onto the scientific field for means of assessment and evaluations of researchers and institutions, they do not come without an impact on the institutional incentives individual researchers face. “There is a powerful feedback between the ranking systems used to assess scientific productivity and the actions of scientists trying to further their careers via these ranking systems” (Braun et al. 2010, 871). In short, the introduction of journal rankings for purposes of evaluation causes reactive behavior invoked by the “authority of numbers” in the sense of Espeland and Sauder, who argue that “[c]asting reactivity as a threat to validity of measures draws attention to the dynamic nature of quantitative authority and its capacity to intervene, sometimes dramatically, with other forms of authority” (Espeland & Sauder 2007, 6-7).

The notion of reactivity is a well-known concept in the methodology of the social sciences and refers to instances in which the application of some instrument of investigation triggers a change in the behavior of the investigated subjects or institutions. Such behavioral changes are conceived as a source of bias, which may arise in the course of data-acquisition (especially in circumstances, where data is collected repeatedly). While it is methodologically correct to speak of a bias here in the sense that such a behavioral change is to be considered as an artifact created in the course of observation, it seems important to note that the change in behavior is, eventually, real: although induced by external observation, such changes in behavior can be profound and have far-reaching effects.

In the case of methodologically guided evaluations in general and evaluative scientometrics in particular, reactivity induces actors to anticipate evaluation criteria. This anticipation in turn affects the behavior of the subjects or institutions evaluated according to these criteria (Ferraro et al. 2005, Ghoshal 2005). Such forms of “evaluation bias” have been confirmed empirically, especially for evaluation procedures such as rankings, where Espeland & Sauder’s (2007) seminal study on the reactivity in the context of the US News’ law school ranking stands out (see also Rafols et al. 2012; Willmott 2011). Another example are international rankings based on indexes such as the World Bank’s Doing Business index or even the prominent Human Development Index used by the United Nations Development Programme (UNDP). Such country rankings may induce reactive behavior of government institutions in favor of the defined ranking criteria.

Whenever the scores of international index rankings are taken literally, the indexes may be poor guides for policies as each link between indicators and scores is noisy and uncertain, but presented as certain. … The belief in accuracy in the presence of inaccuracy, however, may lead to a shift in focus among reformers from what really counts to what the makers of these rankings count. This is the tyranny of international index rankings. (Høyland et al. 2012, 12)

Therefore, it does not come as a surprise that journal rankings do trigger similar effects when used for large-scale institutional evaluation: Italy, for instance, has recently introduced a quantitative standard for supporting decisions on appointment and promotion within the academic system. While the official purpose of this reform is to serve as an objective way to decrease the amount of nepotism in hiring, one cornerstone is the number of scientific publications as indexed by either Scopus or Web of Science. This has led to various attempts of Italian scholars to have past publications that are not covered by Scopus/WoS included into Scopus/WoS and created a strong incentive to only publish in journals covered by these two indices (Abatemarco & Dell’Anno 2012). While we observe an overall “climate of perverse incentives” (Edwards & Roy 2017, 51) and also more blunt forms of reactivity taking the form of simple gaming strategies in academia (Kapeller 2010; Wilhite & Fong 2012; Necker 2014), in this paper we want to ask a more nuanced question: how would the introduction of journal rankings and their rise to institutional authority change existing patterns of scientific recognition to impact citation behavior in economics? Taking the classic Mertonian view (1968) that the future recognition depends on past recognition already received (see also Haucap & Muck 2015), we argue that the introduction of journal rankings as evaluative devices affects the self-reinforcing routines governing the distribution of academic attention by raising the relative importance of journals as compared to specific authors or contributions. As journals gain in visibility in the course of the introduction and utilization of journal rankings, some attention is redistributed from the actual producers and products of scientific activities to their (most important) outlets. And, indeed, Attema et al. claim to have found that U.S. economists “would sacrifice more than half a thumb for an AER publication” (Attema et al. 2014, 495).

So far, we have introduced two hypotheses on reactivity, namely that the introduction of journal rankings redistributes prestige in the favor of journals (see also Mcdonald & Kam 2007; Lariviére & Gingras 2010) and that economists intuitively and/or deliberately consider this redistribution of prestige, which eventually leads to a change in recorded citation patterns. Specifically, our theoretical argument implies that we should see the share of citations received by the top fraction of articles within a top journal decrease, while the share of citations going to the least considered fraction of papers is expected to rise as all papers in a given outlet draw on the latter’s prestige (see fig. 4 for a stylized illustration of the expected effect).

Figure 4. Expected effect of a relative increase in journals’ prestige on observed citation patterns.

We would assume to find effects as envisaged in fig. 4 – an increase in the share of the bottom two thirds of articles with a corresponding decrease in the share of top-cited papers – across all disciplines. However, we also expect these effects to be most pronounced in economics as we assume a higher degree of conformity to institutional incentives in economics as compared to other fields.

In what follows, figs. 5 to 7 show our empirical findings on this question. In acquiring these results we analyzed the citation history of all (top-)journals sampled so far⁸ and further extended our sample by acquiring corresponding data for Nature and Science. More specifically, we compare the distribution of citations to papers published between 1981 and 1985 (counting received citations up to 1990) to the distribution of citations to all articles published between 2004 and 2008 (counting citations up to 2013) for all analyzed journals to identify changes in aggregate citation patterns.

Figure 5. Change of citation patterns in top journals: 1980s vs. today. (The authors’ own graph is based on data from Clarivate Analytics.).

Figure 5 shows the results so obtained by providing individual estimates for all disciplines under study. The resulting pattern closely corresponds with our theoretical expectations – a decrease in the share of citations dedicated to the upper limit of the distribution is complemented by a corresponding increase in lower deciles. The discipline of economics indeed shows the biggest effect with an effective decline in the share of the top decile of papers by roughly nine percent. Only the generalist top journals Science and Nature (as shown in fig. 6) as well as top journals in psychology come close to this pattern, while the same effect for the other social science journals – as well as the physics journals under study – is much less pronounced.

Figure 6. Citation patterns in the 1980s and today: An aggregated comparison. (The authors’ own graph is based on data from Clarivate Analytics.).

Note: Published document types such as editorials, book reviews, bibliographies or biographic items were excluded in this analysis.*Includes sociology, political science and psychology (multidisciplinary).**Includes physics multidisciplinary, physics nuclear, physics atomic molecular chemical, physics condensed matter, and physics particles fields.***The Web of Science Category “Multidisciplinary Sciences” Includes journals that cover a broad spectrum of different specialties, such as PLoS ONE, PNAS, Nature, Science etc… The overlap with other categories (e.g. social sciences, physics) is marginal in both periods (8.1 percent and 3.1 percent of all papers respectively).

Figure 7. Publication output in the 1980s and today: An aggregated comparison. (The authors’ own graph based on data from Clarivate Analytics.).

Figures 6 provides a different perspective on the same data by further aggregating results across disciplines: it shows, that the expected pattern is much more pronounced in economics as compared to the average social science or physics papers in our sample. Indeed, the journals coming closest to the pattern found for economics are multidisciplinary journals, which enjoy an high visibility and, hence, command a considerable amount of prestige. Thus, it seems that, in terms of reactivity induced by journal prestige, economics really plays in the league of the world’s most prolific journals, while leaving the mundane world of ordinary research well behind.

Eventually, the question is what to make of this result? Although the alignment between expected citation behavior and actual patterns retrieved is close and the supposed pattern non-trivial, how can we assure that our inter-temporal comparisons are due to the role of journal rankings and not, let’s say, due to other factors, like different research cultures⁹ or simply the increasing size of the academic literature in general? While such alternative explanations are indeed appealing and, at the same time, the possibilities to quantitatively assess qualitative differences across fields are limited, we can confront the most obvious argument in this context, namely the growth of the academic literature in general.

Figure 7 compares the overall output of scientific publications between the two periods, which has increased significantly in all disciplines under consideration. However, the similarity between the citation patterns in economics and the multidisciplinary journals shown in fig. 6 stands somewhat in contrast to different growth rates in these disciplines (+187 percent in economics vs. +51 percent in multidisciplinary sciences). Conversely, the two remaining disciplines, which experience growth rates more similar to economics (+88 percent in the social sciences and +103 percent in physics), both exhibit a similar citation pattern (see again fig. 6). This pattern is, in turn, quite different to the one in economics. As this comparison is inconclusive, it can hardly be argued that the specific differences noted in figs. 5 and 6 are solely due to the different growth rates affecting these fields.

While our argument – that the difference in responsiveness to incentives across disciplines contributes to the differences in citation patterns – is surely of a cognitive origin, an evaluative approach to scientometrics would be able to offer several alternative explanations. For instance, one could argue that the average quality of papers in top journals has increased more strongly over time than the quality of top-cited papers in these journals – and that the very same process is going on faster in economics than in other fields. Again, the interpretation of the observed empirical patterns depends on our preferred theoretical framing as the data hardly put us in a position to assess the relative appropriateness of a particular perspective.

4. Diversity

Our final section deals with an alleged lack of diversity in economics. Diversity is a concept that can be approached from a variety of angles, as, for instance, demographic, disciplinary, or theoretical diversity. A lack of diversity in terms of demography is well documented in economics (e.g. Bayer & Rouse 2016). Fourcade et al. recently studied the matter of intellectual diversity in economics from a disciplinary angle (Fourcade et al. 2015). By analyzing both, attitudes towards interdisciplinary research as well as interdisciplinary citation patterns in economics, sociology, and political science, Fourcade et al. show that economists are comparatively less inclined to import findings from other disciplines and also have less trust towards interdisciplinary approaches and research strategies (ibid.). These findings are highly consistent with past results from data-driven research on citation patterns, which document the solitary character of economics in terms of disciplinary location and age of imported references. Relative to other disciplines from the natural and the social sciences, papers in economics refer with a higher probability to their own specialty (see fig. 8) and the cited contributions are, on average, older than papers cited in other disciplinary contexts (Wallace et al. 2012). The only discipline in our data exhibiting a higher degree of self-referentiality than economics – astrophysics and astronomy – is coined by a very specific and clearly separable research focus (“outer space”), which may explain the high share of inward-looking citations in this field.¹⁰ For economics, on the other hand, these results not only point to the comparatively strong imprint of traditional approaches and findings on current research, but also signify a comparatively strong intellectual autarky in current economic discourse.

Figure 8. Citation patterns in economics and other disciplines: a large scale sample. Authors’ own graph based on data from Wallace et al. (2012).¹¹.

In a similar vein, also Fourcade et al. (2015) summarized their findings under the heading of “insularity” to signal an absence of theoretical and conceptual diversity. And indeed, economists often conceive interdisciplinary interaction as a competition rather than cooperation, where disciplines are to be judged with regard to their relative “performance.”

“What is interesting to think about are the terms of trade between economics and all these other disciplines. We are clearly a net exporter to political science and sociology” (Wolfers 2010, 30).

In what follows, we want to add to this picture by supplying a perspective on conceptual diversity in economics. This question is especially peculiar as economics can be regarded as a “contested discipline” (Lee et al. 2010, and Lee & Elsner 2011). The character of scientific discourse in economics differs from that found in other social sciences, because economics is dominated by a single paradigm – neoclassical economic theory – which significantly shapes academic teaching and economic research.¹² A thus resulting exclusion of alternative economic approaches, like institutional microeconomics, evolutionary analysis of economic change, or Keynesian macroeconomics, from the “official” economic discourse in further consequence led to the foundation of – nowadays eminent – heterodox economic journals, like the Cambridge Journal of Economics, the Journal of Post Keynesian Economics or the Journal of Economic Issues in the late 1960s and 1970s (King 2003, 134-136). Notwithstanding these efforts for providing alternative economic approaches with adequate institutional foundations, alternative or “heterodox” schools of thought nowadays only constitute a small fraction inside the economics discipline. While heterodox economists are confronted with a series of exclusionary practices and routines (Dobusch & Kapeller 2009; King 2013), their works also draw on a broader variety of sources and inputs and, hence, exhibit a distinctive, more interdisciplinary communicative pattern (Glötzl & Aigner 2015). Moreover, Glötzl & Aigner (2015) provide additional evidence for this contestedness of economics by pointing to a “mainstream core – heterodox periphery structure” (see, also, Dobusch & Kapeller 2012a) in the citation networks associated with different economics departments at the University of Vienna and the Vienna University of Economics and Business, that differ with respect to their openness for alternative and heterodox approaches in economics.

The resulting lack of diversity within the economic discipline has been studied empirically, in particular by analyzing the relative citation flows between 13 high-ranked mainstream and 13 high-ranked heterodox economic journals in two different periods: before 1989-2008 (Dobusch & Kapeller 2012b) and after the recent financial and economic crisis, 2009-2013 (Aistleitner et al. 2018). The results indicate that the monistic and closed citation behavior exhibited by (highly-ranked) mainstream economic journals remained stable after the crisis, as the mainstream journals continue to devote only minimal attention to heterodox inputs (as is evidenced by the small share of citations – less than 3 percent – imported from heterodox sources¹³ – see fig. 9). On the contrary, the heterodox journals themselves exhibit a more balanced citation pattern with import shares from the mainstream sample of 47.6 percent (pre-crisis) and 24.6 percent (post-crisis), where this decrease in part also reflects the fact that heterodox contributions typically cite older papers from the mainstream literature more strongly than recent ones. In addition, this result lends support to Kuhn’s 1962 prediction, which stated that a dominant paradigm that faces a significant anomaly (the crisis) will strive for adapting already established theses, models, and methods instead of turning to more fundamental revisions.

Figure 9. Diversity and the interaction pattern between economic mainstream and heterodox journals. (a) the discourse between a sample of top 13 orthodox and top 13 heterodox journals (1989-2008; taken from Dobusch & Kapeller 2012b). (b) replication with a corresponding control group (1989-2008; the authors’ own calculation based on data from Clarivate Analytics). (c) replication for the post-crisis period (2009-2013; the authors’ own calculation based on data from Clarivate Analytics). (d) Analysis of a large-scale journal sample (1969-2013; taken from Aigner & Glötzl, in this issue).¹⁴

To further assess the robustness of these results, fig. 9 additionally shows results derived from data on a control group and a large-scale sample. The first robustness-check uses a control group to ask whether the comparative neglect of heterodox inputs in mainstream outlets can be explained by the lower ranks of heterodox journals, which supposedly signify a lower overall quality. In composing the control group we used a group of mainstream journals with similar ranking positions than the heterodox sample. The second robustness check analyzes data of a large-scale sample acquired from the Web of Science database to scrutinize the robustness of the observed differences.

By comparing these results to other recent findings on the “insularity” of the economics discipline (Fourcade et al. 2015), one notices several similarities between an interdisciplinary and an intradisciplinary view on conceptual openness in economics. For once, we find that economics is comparatively closed not only to outside disciplines, but also to theoretical outsiders within the discipline. For another, we observe that these theoretical outsiders share close ties to other branches of the social sciences (e.g. Glötzl & Aigner 2015), and thereby do not resemble the disciplinary isolation practiced by the economic mainstream. A possible, tentative conclusion to be drawn from this is that the interdisciplinary insularity of economics as diagnosed by Fourcade et al. (2015) or Gingras and Schinckus (2012) with respect to econophysics is not primarily due to the idiosyncracies and peculiarities of the subject matter in economics, but rather represents an outgrowth of the dominant economic approach which seems to discount external inputs a priori.

Qualitative examples for illustrating the above line of argument are easily found. One instance is provided by the “Economics of Identity” – a stream of research based on a well-received paper by Akerlof & Kranton (2000), which gathered more than 800 citations in Web of Science, most of them from the field of economics. In this context, it is interesting to note that the conceptual core of “identity” as introduced in economics – that people have ideas about who they are and try to conform to these ideas in their behavior – overlaps strongly with the meaning of identity as used by social psychologists since, at least, the 1950s. While Akerlof & Kranton (2000) obviously import this idea into economics and thereby attract much interest within their profession, classic contributions on social identity in other fields, e.g. Tajfel (1974) or Ashforth & Mael (1989), remain largely neglected in the economic discourse, although they are widely cited in general. In sum, these two papers have more than 3,000 documented citations in Web of Science, where only 34 of those stem from economics.¹⁵ Hence, Akerlof & Kranton (2000) have contributed significantly to increasing the prominence of the psychological concept of identity within economics. It, however, remains difficult to say, whether this increase in prominence is to be attributed to an increase in visibility or legitimacy. In the former interpretation, “economists read narrowly” and Akerlof & Kranton (2000) simply introduce economists to a hitherto unknown concept – personal identity – that is in turn taken up for further investigation. In the latter interpretation, “only economists do it right” and personal identity as a theoretical concept gains credibility and plausibility exactly because it is being taken up by well-known economists in a very respected outlet. While our results point towards the importance of both factors, visibility as well as legitimacy, it is remarkable that the integration of identity in the economics’ discourse does not spill over to the extra-disciplinary origins of said concept.

Against this backdrop, the relative “superiority” of economics derived from inspecting citation-trade balances – as suggested by Wolfers (2010) – can be explained not only by a higher quality of research outputs, but also by a higher degree of group-think and inward-looking behavior within the economics profession, which effectively suppresses the import of ‘foreign’ citations. As we have seen, such an interpretation would be well in line with the average economists stance on the value and merit of extra-disciplinary contributions. Yet again, we find that the two ways of looking at and interpreting citation data lead to very different implications.

5. Conclusion

Most scientometric studies up to now remain committed to a certain perspective – either cognitive or evaluative – which then frames our understanding of the relevant data and has a strong imprint on the interpretation of results and the lessons drawn from these outcomes. In this paper we use the peculiar case of economics as an example for illustrating the partially stark differences arising from these two different points of view.

We have shown how citation patterns in current economics systematically differ from patterns found in other disciplines and thereby further substantiated past results regarding the peculiar disciplinary character of academic economics (e.g. Dobusch & Kapeller 2009; Fourcade et al. 2015; Leijonhufvud 1973; Morgan 1988). We suggested explaining these differences with reference to specific characteristics of economics, namely that economics exhibits a more intense disciplinary stratification (leading to and reinforcing concentration of attention within economics), that economists show greater awareness for institutional incentives (fostering reactive citation behavior) and, finally, that economics is a contested discipline dominated by a single approach (resulting in a lack of openness for external inputs).

These observations thereby align well with core elements of the dominant theoretical approach in economics: in the standard textbook view, competition is objective and leads to superior results when compared to other modes of social organization – hence, a focus on top journals seems natural and legitimate. Similarly, textbooks portray deviations from instrumental rationality and a corresponding neglect of incentives as somehow foolish behavior – hence, reactivity can be easily framed as a rational response to given constraints. Finally, the standard textbook view suggests that scientific progress has, more or less, linear properties – hence, a single, dominant theoretical approach is seen not as a limitation for the field, but, quite on the contrary, as a signifier for the superior status of economics as compared to its neighboring disciplines.

While we found that the data does really lend some support to our main hypotheses, all of our findings can be subjected to different interpretations. Higher concentration of attention might be due to more precise instruments for detecting the quality of manuscripts before publication. What seems like reactivity might be emerging from an otherwise invisible shift in patterns of quality in the underlying literature. And openness and ignorance in economic thinking might simply reflect differences in quality located at the level of more fundamental ideas. Our main message in this regard is to think carefully about one’s preconceptions on what citations really signify, because, as we have shown, these preconceptions eventually determine the implications drawn from seemingly innocent empirical outcomes and results.

Finally, our work also points to several policy implications regarding the link between current practices in research evaluation and the development of the economic discipline. In line with earlier findings on the disciplines’ “contestedness,” our results suggest that the incentives arising from imposing citations as the “currency of our industry” (Coffman et al. 2017, 3) not only reinforces the risk of marginalizing alternative theoretical approaches within the discipline. It also tends to hamper intellectual exchange of knowledge across its disciplinary borders.