As evidence-based medicine is standard of care and peer-reviewed publications have gained more value, physician-scientists are constantly working to publish work that has clinical impact. This drive to publish important knowledge is done for altruistic reasons as well as for career advancement. Academic promotion for physician-scientists is often based on criteria such as number of peer-reviewed publications, authorship order, journal Impact Factor, grant funding, number of citations, and supposed publication quality. Reference Rice, Raffoul, Ioannidis and Moher1 Journal Impact Factor has been used frequently to gauge the academic output of an individual author, but it has limitations in quantifying the quality of a study and varies widely depending on the field of medicine. Reference Loomba and Anderson2–Reference Hill and Jenkins5 The use of number of citations is also limited as it carries the assumption that references are selected due to their quality, when in reality other factors often play a role. Reference Seglen6 Authors often cite secondary sources (not primary publications), copy references, cite themselves or colleagues, do not cite established knowledge, or cite a particular manuscript to gain favour with specific journal editors. Reference Seglen6 It has been demonstrated that measures of the quality of a study, the design of a study, and the use of appropriate statistical analysis do not correlate with the number of citations. Reference Nieminen, Carpenter, Rucker and Schumacher7–Reference Patel, Vaduganathan, Bhatt and Bonow9 Journal prestige and reputation is more frequently found to correlate with citations. Reference Callaham10 Despite their limitations, Impact Factor and citation number are used frequently.
Other newer metrics utilised are reads, downloads, and social media attention (Altmetric score). These are imperfect, as well, as many of these are most likely indicators of manuscript consumption and not necessarily manuscript quality. They may reflect the interest of the scientific community for a specific research question rather than the quality of the study. The Altmetric score is an indicator of the dissemination, attention, influence, and impact an article has on online platforms such as public policy documents, media outlets, Wikipedia, personal blogs, community forums, and social media including Facebook, Twitter, Instagram, YouTube, and LinkedIn. Reference Araujo, Vanin, Nascimento, Gonzalez and Costa11 Some assume that an increased Altmetric score, and thus greater online presence, should have a positive correlation with the number of citations, although previous studies in other specialties have shown mixed results. Reference Ruan, Chen, Cohen, Singhal, Lin and Lee12–Reference Rong, Lopes, Hameed, Gaudino and Charlson18
The primary objective of this study was to determine if Altmetric score, number of reads, and number of citations for paediatric cardiology manuscripts correlate with one another. A secondary objective was to determine the extent to which factors mediated citation number for paediatric cardiology manuscripts.
Methods
Data source
Data for this study came from manuscripts published in Cardiology in the Young from 2010 through February 2021. Data were sourced from this singular journal as the Altmetric score, number of PDF reads, and citation number were readily available. No other paediatric cardiology journal has readily available data as Cardiology in the Young. These data are publicly available on the journal’s website. The data appear on the page for each specific study. No additional source of data such as Web of Science or Google Scholar was utilised.
Data extraction
Manuscripts published in the journal during the study period were identified by the use of the digital archives on the journal website. Manuscript-specific data were then extracted by using the publicly available data regarding number of reads, number of citations, and Altmetric score shared on the journal website itself. These data were extracted from 3 March to 25 March, 2021 by two separate authors (DS and CA) independently. Year of publication and manuscript type were also collected. A third author (SF) then reviewed the independent data extraction and identified any discrepancies. Discrepancies were then reviewed by two separate authors (EV and JF) with a consensus achieved. Manuscript type was assigned as it was by the journal itself as one of the following: brief report, original study, images in CHD, or review. Editorials were not included in these analyses.
Statistical analyses
Normalcy of distribution of reads, citations, and Altmetric score was conducted by evaluation of skewness and kurtosis. As there was a non-normal distribution of data for all three of these, non-parametric statistical tests were used. Spearman’s correlation analyses were conducted between manuscript reads, citations, and Altmetric score. Correlation analyses were done for all manuscripts combined and then for each manuscript type, separately. Correlation analyses were also repeated for the manuscripts with the most citations. Specifically, this meant manuscripts in the top 10th percentile for citation number. This was an arbitrarily selected cut-off. An r-value of 0.90 to 1.00 was considered to represent a very strong correlation, 0.70 to 0.89 a strong correlation, 0.50 to 0.69 a moderate correlation, 0.30 to 0.49 a poor correlation, and less than 0.30 a negligible correlation.
Next, a regression analysis was conducted to model citations. A stepwise regression was conducted with number of citations as the dependent variable and year of publication, publication type, number of reads, and Altmetric score as independent variables. Citation number was selected as the dependent variable as this is a frequently cited metric of publication performance. All statistical analyses were conducted using SPSS Version 23.0. A p-value of less than 0.05 was considered statistically significant.
Results
A total of 2642 manuscripts were included in the final analyses. Of these, original studies represented a majority (Fig 1). The number of manuscripts published each year is outlined in Figure 2.
Figure 1. Types of studies.
Figure 2. Number of manuscripts published each year.
Correlation of reads, citations, and Altmetric score
When all studies were analysed together, reads, citations, and Altmetric score all demonstrated statistically significant correlation. Reads and citations had poor correlation (r-value 0.32), reads and Altmetric score had negligible correlation (r-value 0.26), and Altmetric score and citations had negligible correlation (r-value 0.07) (Table 1). Correlation between these metrics was not greater in any single subset of manuscript type. Results of these correlation analyses are outlined in Table 1.
Table 1. Spearman’s correlation coefficient and p-value of correlations between the various metrics for all studies and specific study types
* A p-value of less than 0.05 was considered statistically significant.
When only manuscripts in the top 10th percentile for citation number were included in correlation analyses, reads and citations had negligible correlation (r-value 0.27), reads and Altmetric score had poor correlation (r-value 0.30), and Altmetric score and citations had negligible correlation (r-value 0.18). Results of these correlation analyses are outlined in Table 2.
Table 2. Spearman’s correlation coefficient and p-value of correlations between the studies with the top 10th percentile for citation number
* A p-value of less than 0.05 was considered statistically significant.
Regression analysis for number of citations
Regression analysis with number of citations as the dependent variable demonstrated significant associations between number of citations and year of publication, manuscript type, number of reads, and Altmetric score (Table 3). Year of publication was independently associated with number of citations (beta-coefficient –0.95, p-value < 0.01), indicating that older manuscripts tended to receive more citations. Manuscript type was independently associated with number of citations (beta-coefficient 1.04, p-value < 0.01), indicating that reviews tended to receive more citations. Number of reads was independently associated with number of citations (beta-coefficient 0.01, p-value < 0.01), indicating that more read manuscripts tended to receive more citations. Altmetric score was independently associated with number of citations (beta-coefficient 0.05, p-value < 0.01), indicating that manuscripts with higher Altmetric scores tended to receive more citations.
Table 3. Stepwise regression analysis to determine impact of variables on citation number
* A p-value of less than 0.05 was considered statistically significant.
It is important to reconcile our findings that the correlation between the various metrics produced negligible or poor correlations, but they are still statistically significantly related. Statistical significance, as defined by p-value, simply indicates that the null hypothesis can be rejected, with the likelihood of a falsely rejected null hypothesis being equal to the p-value. The p-value does not indicate anything about the magnitude or direction of the association. Two metrics can be linked in a significant manner, but their correlations do not have to be strong. In this situation, a statistically significant coefficient may exist, but no true practical, applicable significance exists. Further explanation of this concept can be found in pertinent literature. Reference Schober, Boer and Schwarte19
Discussion
Academic publishing amongst physician-scientists continues to grow. Academic output in the form of publishing has now become a large piece of academic promotion for academic physicians. Reference Rice, Raffoul, Ioannidis and Moher1 As publishing has become associated with promotion, there are now a variety of objective metrics that are utilised to determine “performance” or “quality” of published manuscripts. Reference Aksnes, Langfeldt and Wouters20 These include reads, downloads, citations, and social media attention. Reference Ladeiras-Lopes, Clarke and Vidal-Perez21,Reference Hong, Yoon and Lim22 This study compared reads, citations, and Altmetric score for manuscripts published in Cardiology in the Young during a 11-year period to determine if there is any correlation between these metrics. These various metrics all had, at most, weak correlation with one another, indicating that these metrics represent discordant phenomena. Reads, citations, and Altmetric score seem to capture different aspects of manuscript utilisation.
Reads and citations had poor correlation (r-value 0.32). This may be for several reasons. First, only a subset of those reading a manuscript will be publishing a manuscript. An even smaller subset of readers who will go on to publish a manuscript is likely to publish regarding similar enough subject matter to warrant a citation of the initial manuscript. Thus, it is simply more likely that most readers simply will never publish a manuscript in which they could potentially cite another paper. Most readers likely represent those reading the manuscript purely out of intellectual curiosity and/or for potential application to clinical practice. Additionally, those who may be publishing on the same subject matter may also find a paper not to be relevant to cite in their own manuscript.
Reads and Altmetric score had negligible correlation (r-value 0.26). This is particularly interesting as many social media posts regarding academic publications tend to have hyperlinks to the manuscript itself. The discordance between Altmetric score and reads highlights potential differences in conventional manuscript consumption and social media manuscript consumption. First, social media and conventional manuscript reads likely capture a different group of readers. Altmetric score is heavily based on Twitter, and thus, the Altmetric score may capture a demographic that is savvier with social media, such as younger readers. Additionally, the discordance between reads and Altmetric score also highlights the possibility that those being exposed to manuscripts through social media platforms may not actually be reading the manuscript in its entirety. Thus, these consumers may only be exposed to summative infographics or text regarding key findings. A potential conventional correlation to this is those who simply read the abstract for manuscripts in print.
The regression analysis, modelling for number of citations, demonstrated that there was an independent association between number of reads and citations as well as Altmetric score and citations. For every 100 PDF reads, there was an increase in the number of citations by 1, and for every 20 increase in the Altmetric score, there was an increase in the number of citations by 1. Put in an alternate way, there was a 100:1 read to number of citation ratio and a 20:1 Altmetric score to number of citation ratio. So, while these were statistically significant, the actual contribution of these from a practical standpoint seems to be less significant. The regression analysis further highlights the independence of number of reads and Altmetric score. These two did not demonstrate significant collinearity when entered into the regression analysis and simultaneously demonstrated statistically significant associations, independently.
So then, what do these findings mean? First, reads, citations, and Altmetric score all have weak correlation. Even when we compare the top 10th percentile, the correlations are weak. This means that the correlations hold true across variously performing studies. They seem to represent discordant phenomena when it comes to manuscript consumption. Some composite of these metrics likely will ultimately offer the best insight into manuscript consumption. Each metric also may capture a different demographic of manuscript consumers as well. It is important to keep in mind that these three metrics are simply representative of various forms of consumption of manuscripts. These three metrics are not inherently metrics of quality. Quality can be defined in various ways by different individuals: robustness of study methodology, reproducibility of findings, clarity in reporting results, effectiveness in framing the results regarding clinical relevance, number of reads, number of citations, and amount of social media mentions.
As these are all different metrics, it becomes clear that when an individual attempts to comment on the “quality” of a manuscript, it is a highly ambiguous and subjective endeavour. Often when the topic of “quality” arises with respect to a manuscript, “quality” is never actually defined. Assessment of publication “quality” is an inherently flawed concept. Multiple elements contribute to the futility of attempting to quantify the quality of a manuscript: lack of a universal definition, lack of clarity regarding the actual definition being used when “quality” is being discussed, consumption metrics being used to judge “quality,” the limitations of current bibliometric indices, and the subjective nature of the notion of quality. The difficulty associated with quantifying the quality of a manuscript is problematic because academic promotion is often based on the notion of publication quality. Each promotion committee will have a different means by which academic output and published manuscripts are judged. Anecdotally, a majority of these will select one or two of the aforementioned metrics, all of which have significant limitations. Many of these bibliometric indices have also not been described in detail for specific disciplines. Available data are also, anecdotally, poorly understood by those who use them for determining promotion. As published manuscripts are often used for academic promotion, the most used metrics in this regard, anecdotally, are journal Impact Factor for journals published in total number of publications, total citations for all of an author’s publications, and author Hirsch index. Reference Rice, Raffoul, Ioannidis and Moher1 Obviously, each individual metric is flawed in judging the “quality” of the academic publications of a physician-scientist.
Journal Impact Factor is a citation-based metric which considers the citation impact of papers within a given time frame. It is calculated by dividing the number of citations in a given year to papers published in the previous two years by the number of citeable items published in the previous two years. For example, the 2020 Impact Factor is calculated by dividing the number of citations in 2020 to content published in 2018 and 2019 by the number of citeable items published in 2018 and 2019. Journal Impact Factor is nearly a nonsensical method for evaluation of the quality of a given manuscript because it has nothing to do with the individual publication but instead is an aggregate metric for the journal itself. Reference Loomba and Anderson2 Certainly, if one was to ask another what their height is, receiving the average human height in response does not answer the question of the height of the individual. Similarly, journal Impact Factor has nothing to do with the performance, nonetheless “quality,” of a given manuscript. Reference Leff3
Total number of publications is limited because studies require different amounts of resources, whether it be labour or financial support. Simple number of published manuscripts does not do anything to capture complexity. Total citations for all of the publications of an author are limited because citation potential varies from discipline to discipline based on the number of physician-scientists in the discipline and the publication output of these physician-scientists in the discipline. Reference Epstein23 While this concept seems relatively intuitive, it is rarely ever acknowledged. Certainly, most disciplines in medicine have not robustly defined this metric for themselves. Of note, this has been done in paediatric cardiology. Reference Loomba and Anderson2,Reference Barakat, Nimri and Shokr24 A randomised trial of how social media exposure improves impact metrics in a renowned cardiovascular journal (Circulation) concluded that social media strategies do not increase the number of views or reads in this journal. Reference Fox, Bonaca, Ryan, Massaro, Barry and Loscalzo25 Citation count is also limited as it does not actually speak to the clinical applicability of the data presented in the manuscript. Reference Hanney, Home, Frame, Grant, Green and Buxton26 Consequently, citations in clinical guidelines have been proposed as a better indicator to measure the clinical impact of an article. Reference Grant27,Reference Lewison and Sullivan28 Citation count may be impacted not only by the number of people with interest in paediatric cardiology but also by the number of people working on specific topics and research fields.
Other metrics that are based on citation count, such as Hirsch index, have the same limitations as citation count, but also have additional limitations imposed by the additional factors included in the specific metric. Reference Waltman and van Eck29 The Hirsch index is also known as the h-index and is designed to improve upon simpler measures such as the total number of citations or publications. The h-index is defined as the maximum value of h such that the given author/journal has published at least h papers that have each been cited at least h times. In other words, the h-index is the largest number h such that h articles have at least h citations each. For example, if an author has five publications, with 9, 7, 6, 2, and 1 citations, respectively, (ordered from greatest to least), then the h-index of the author is 3, because the author has three publications with 3 or more citations. However, the author does not have four publications with 4 or more citations.
These findings of discordance between these bibliometric indices are not novel. Similar analyses have been done in other disciplines of medicine. Reference Chen, Bukhari, Cockshull and Galloway30–Reference Kampman, Hermanides, Boere and Hollmann33 This manuscript, however, reports the first such analyses done in paediatric cardiac care. Bibliometric indices require discipline-specific description. Indices will have different values and meanings in the context of various disciplines. Each individual metric must also be interpreted in the context of its own limitations. It must also be kept in mind that bibliometric indices all represent a very specific phenomenon of manuscript consumption. Various bibliometric indices complement one another. Using one bibliometric index over another with a sense of one being superior to the other is ill-founded, as there is poor a correlation between various bibliometric indices. A combination of metrics should be used to truly develop a more comprehensive evaluation of manuscripts. The exact methodology of such an evaluation is beyond the scope of this study and the data in this current study.
Conclusion
This study describes the correlation of reads, citations, and Altmetric score in manuscripts published in Cardiology in the Young, demonstrating poor correlation, at best, between these metrics. Each bibliometric index seems to represent a different phenomenon of manuscript consumption. No single bibliometric index in isolation offers ample representation of manuscript consumption, let alone manuscript quality.
Acknowledgements
None.
Financial support
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Conflicts of interest
None of the authors have any pertinent conflicts of interest to disclose.
Ethical standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national guidelines on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
