Open versus blind peer review: is anonymity better than transparency?

Summary

Peer review is widely accepted as essential to ensuring scientific quality in academic journals, yet little training is provided in the specifics of how to conduct peer review. In this article we describe the different forms of peer review, with a particular focus on the differences between single-blind, double-blind and open peer review, and the advantages and disadvantages of each. These illustrate some of the challenges facing the community of authors, editors, reviewers and readers in relation to the process of peer review. We also describe other forms of peer review, such as post-publication review, transferable review and collaborative review, and encourage clinicians and academics at all training stages to engage in the practice of peer review as part of continuing professional development.

LEARNING OBJECTIVES

After reading this article you will be able to:

• • describe the commonly used forms of peer review

• • understand the main advantages and disadvantages of each type of peer review

• • feel confident engaging in peer review, whether pre- or post-publication.

Peer review in scientific journals is an established method of ensuring quality in academia. It has existed in a form that we would recognise since the 1800s, and since the early 20th century has been used as a form of gatekeeping to help decide which articles should be published (Csiszar 2016). Although guidance has previously been published in this journal on the principles of peer reviewing (Halder 2011; 2020), there is little guidance available comparing the different forms of peer review from the perspectives of authors and peer reviewers of psychiatric journals and their readership. These forms include single-blind, double-blind and open peer review. An understanding of the process of peer review is important, as it helps to decide whether a published article meets strict criteria for academic rigour, which journal to submit a manuscript to and which requests for peer review to accept.

Peer review has two main aims: to assist journal editors in decision-making regarding publication of articles and to help authors improve the standard of their work (Halder 2011). Modern-day peer review has been described as a process in which research submissions are ‘reviewed by a committee whose membership has the expertise to provide optimal critical evaluation and feedback and is free of conflict or bias’ (Liaw 2017). It is regarded as a key component of the scientific process, and is critical to establishing and maintaining a journal's reputation and impact factor (Halder 2011; Largent 2016).

Despite peer review being well-established in academia, there is no consensus as to what form it should take. Consequently, journals differ in their approach (Godlee 2002; Tomkins 2017; Haffar 2019). Currently, the most common type employed is single-blind peer review (Wiley 2020), in which the author's identity and institution are visible to each reviewer, but each reviewer's identity is not known to the author. An alternative approach is double-blind review, in which neither authors nor reviewers are aware of each others' identity. More recently, the approach of open peer review, where both authors and reviewers are known to each other, has gained traction. The advantages and disadvantages of these and other types of peer review are discussed below.

Single-blind peer review

This is where reviewers know the identity of authors, but authors do not know the identity of reviewers.

It is an approach used by journals such as the Journal of Psychiatric Research, Acta Psychiatrica Scandinavica, The Lancet Psychiatry, Archives of Suicide Research, The British Journal of Psychiatry (BJPsych) and BJPsych Open.

Advantages

One advantage of single-blind peer review is that hiding reviewers’ identities might lead them to feel able to appraise an article with greater honesty, unfettered by potential sensitivities. For example, newer researchers might be concerned about damaging their career opportunities if they criticise the work of a more senior author during peer review. Providing anonymity removes this barrier and allows for a more honest, and potentially more constructive, review (Godlee 2002; Haffar 2019; Wiley 2020). Journal authors sometimes struggle to identify reviewers for manuscripts submitted, and if blind peer review is a factor encouraging participation from academics at all career stages, then this makes it an attractive option to all parties.

There are also advantages in providing the identity and affiliations of authors to reviewers. It can lead to a more contextualised review, in that the findings reported are appraised in the light of the work of the group carrying out the study (Tomkins 2017). It also makes it easier for reviewers to identify conflicts of interest that arise where they have previously worked with the authors, allowing them to turn down offers to review when this creates the potential for biased review (Tomkins 2017). In addition, knowing the authors’ identities could help newer reviewers who are learning about their field to gain familiarity with the work of this research team. They may even consider contacting them to suggest future collaboration once the journal's final decision has been made.

Disadvantages

One of the main disadvantages of single-blind review is that it may allow for discrimination on the basis of attributes other than scientific merit (Godlee 2002; Haffar 2019; Wiley 2020). Such attributes include gender, ethnicity, experience or academic reputation, whether of the authors or their institutions. This might be a particular problem for authors from countries where the primary language is not English, as geographical discrimination can easily be hidden under the guise of poor language (Horton 2003; Cox 2019; Pitman 2019). It has been suggested that even a perception of bias in favour of seniority, gender or ethnic group may be discouraging for early career researchers and it is therefore important to address whether or not true bias exists (Snodgrass 2006).

The potential for bias does not seem to be purely theoretical, and there is clear evidence that single-blind peer review favours famous authors and prestigious institutions (Tomkins 2017; Goues 2018). This prestige bias illustrates the risk that the halo effect of these academics and of well-reputed universities may dazzle reviewers at the expense of noticing methodological problems, errors, lazy citations and over-interpretation of findings.

The finding that single-blind review might lead to greater discrimination against female authors has been called the Matilda effect (Rossiter 1993). This is named after Matilda Gage, an American 19th-century suffragist and feminist critic, and it describes the underrecognition of female scientists. Whether single-blind peer review does facilitate a Matilda effect in academic publishing is unclear. Earlier research on the impact of masking as used in peer review found that female first authorship was 7.9% higher in Behavioural Ecology after that journal switched from single-blind to double-blind reviews, as compared with no increase in the incidence of female first authorship in five comparable ecology journals retaining single-blind review over the same period (Budden 2008).

Later studies have not found that removing the names of authors from papers leads to greater acceptance rates for authors whose names identify them as female (Tomkins 2017; Cox 2019). However, when meta-analysed, this body of literature does find that single-blind review discriminates against female authors (Tomkins 2017). Although there may be evidence that the proportion of female first authorship increased at least in the early stages of the introduction of double-blind reviewing in some journals, this seems to have occurred alongside an overall increase in female authors, and so it is hard to ascertain whether or not the two are associated (Webb 2008).

The finding that single-blind review leads to a lower rate of recommendation for publication seems to be consistent across studies (Tomkins 2017). From an author's perspective, single-blind review might therefore be disadvantageous when considering which journal to submit to.

Double-blind peer review

This is an approach where neither authors nor reviewers are aware of the others’ identities and it is used by journals such as Social Science & Medicine, General Psychiatry and the American Journal of Neuroradiology.

Advantages

The main advantage of double-blind peer review is that it reduces the potential for biased reviews based on views about the authors’ or affiliation institutions' attributes. Early career researchers in particular have been shown to favour double-blind peer review when submitting manuscripts because it reduces the risk of prejudice that might disadvantage younger or less experienced authors, women and ethnic minority authors (Rodríguez-Bravo 2017; Goues 2018). Ultimately, double-blind peer review might be a fairer process owing to its potential to minimise these biases. For reviewers, just as with single-blind peer review, hiding reviewers’ identities frees them to appraise an article with greater honesty, uninhibited by the fear of offending anyone on the authors’ team.

A survey of over 4000 scientists by Sense about Science in 2009 found that the double-blind format is indeed preferable to reviewers, with 76% preferring this option, and may even increase willingness to participate (Sense about Science 2016, cited in Halder 2011). This is consistent with the findings of two surveys conducted by publishing consortia (Ware 2008; Taylor & Frances 2015) and a qualitative study exploring the views of early career researchers regarding peer review (Rodríguez-Bravo 2017). However, a third survey by a publishing consortium, published in 2016, found that reviewers had no preference between double- and single-blind review (Ware 2016). These surveys were typically conducted by emailing thousands of reviewers in their journals’ pool of reviewers. Where response rates are available, these are in the range of 2–10%, so sampling bias seems very possible (Taylor & Frances 2015; Ware 2016; Sense about Science 2016).

Disadvantages

A practical disadvantage of double-blind peer review is that additional time, effort and cost may be required to make manuscripts anonymous. Editors in many fields have traditionally been resistant to double-blind peer review, perhaps for this logistical reason (Webb 2008).

Critics of this approach have argued that double-blinding might be ineffective, since authors can often be identified through their scientific area, citations or writing style. A review of studies that assessed the effectiveness of blinding found that it was successful in an average of 62% of cases, with self-citation being the strongest clue as to authorship (Snodgrass 2006). In one study, reviewers’ ability to guess the author and/or institution was associated with a higher rate of manuscript acceptance (O'Connor 2017).

Double-blind peer review makes it difficult for potential reviewers to identify conflicts of interest, as they will be unaware if they have collaborated with the submitting authors. One software tool available to editors when selecting potential reviewers allows automated detection of professional connections. There is some experimental evidence to support this method in identifying authors and reviewers who have worked together (Tomkins 2017). However, many collaborations (past and planned) may not be apparent online, and no software will be able to identify these.

The other conflict of interest that could remain undiscovered is where authors have not disclosed in their submission the receipt of industry funding. Single-blind peer review could mean that reviewers familiar with the authors’ field might be aware of undisclosed industry ties (Tomkins 2017). However, with double-blind peer review these ties would remain concealed.

Open peer review

This is where both authors and reviewers know each others’ identities, and it is used by the BMJ, BMJ Open, BMC Psychiatry and BMC Psychology.

Some journals now favour open peer review for the transparency afforded by this approach (Halder 2011). Not only are both authors and reviewers aware of each others' identities, but reviews are sometimes published alongside the final article. Journals such as the BMJ argue that the case for open review is ultimately ethical, for putting authors and reviewers in equal positions and for increasing accountability (Smith 1997). Relatively few psychiatric journals use open peer review; exceptions, as mentioned above, are BMC Psychiatry and BMC Psychology. In a large 2015 survey, 50–70% of researchers reported favourable attitudes to open review, though this fell to 35–55% when the process included publishing reviews and reviewer identities alongside the paper (Ware 2016).

Advantages

The major benefit of open review is that it increases the visibility of reviewers, making them more accountable for their comments (Godlee 2002). This may improve the quality of the review and reduce the temptation to suggest that revisions include citations of their own work, except where clearly relevant. It also means that editors are more accountable for their choice of reviewer and the weight they give to each reviewer's views (Godlee 2002). There is evidence that open peer review produces better quality of reviews, which may indicate greater diligence and attention to detail. In a randomised controlled trial, reviewers were allocated submitted papers and randomly assigned to the open or anonymous review groups to compare quality of reviews (Walsh 2000). The study found that the open reviews were of higher quality, were more courteous and took longer to complete than anonymous reviews. The study only randomised reviewers who said at the outset that they would be happy to reveal their names to the authors whose papers they reviewed, and 76% of reviewers were willing to do so.

A further benefit to open peer review is that reviewers can receive recognition for high-quality reviews. Currently, reviewers dedicate a significant amount of time to this task, with relatively little credit (Godlee 2002). Although peer review registration sites exist to collate metrics on completed reviews (see below), some produce raw numbers of reviews by journal, rather than allowing readers to evaluate the quality of the peer review itself.

Open peer review might be preferred when there is significant scope for conflicts of interest, such as in pharmacology trials or journals where industry sponsorship could be a frequent source of reviewer bias (Moylan 2014). At least 70 journals listed in BioMed Central have moved to open peer review (Haffar 2019) and this seems to reflect a gradual shift in biomedical publishing.

Open peer review also allows reviewers to compare their submitted reviews with those of named reviewers, setting their comments in the context of their past work and collaborations. This process of comparison serves as a way of improving a reviewer's research and critical appraisal skills, through seeing how another reviewer approached the same task and which methodological problems each may have missed. Where reviews (and successful resubmissions) are available to readers alongside the article, this also has educational value in helping readers build critical appraisal skills. It may also be instructive in illustrating the appropriate tone to take when responding to reviewers. By setting out the timeline of article submission, review, revision and acceptance, fully open peer review has the advantage of editorial transparency and an insight into the publication process.

Open peer review may offer authors the best chance of publication, given the findings of a randomised trial of open versus single-blind peer review that reviewers who signed their names to reviews were more likely to recommend publication (Walsh 2000). It is not clear whether this was due to feelings of guilt, perverse incentives to please influential authors or whether more thorough review (which was also evidenced in this study) had uncovered the true merits of the paper. Further qualitative research with reviewers would help identify which forms of review are more acceptable to them, and whether the incentive structures inherent to any of these approaches pose a threat to integrity and the quality of published scientific research.

Disadvantages

Where reviewers feel open to wide scrutiny by their peers in conducting an open review, they may seek to be more thorough, thus taking more care and time when completing each review. This is clearly more resource intensive, even where it is also a useful learning experience (Walsh 2000). There may also be sensitivities involved in agreeing to open peer review if the flaws of the manuscript are apparent from the abstract in the invitation to review. This creates the potential for awkwardness in situations where the reviewer knows one of the authors indirectly, but not well enough to present a conflict of interest; or hopes to collaborate with one of the authors in the future. In such cases submitting a negative, albeit constructive, review could engender anxiety on the part of the reviewer that future collaborations might be jeopardised. A study of early career researchers found that many were uncomfortable with the idea of open peer review, with their concerns including a fear of reprisals via social media (Rodríguez-Bravo 2017). Some participants also suggested that they felt unsuitably qualified to criticise their peers. All these factors might reduce willingness to review or create perverse incentives to return artificially positive reviews. However, all authors should value a fair and considered critique of their work and, regardless of seniority, should be able to process constructive criticism from even the most junior trainee.

Other forms of peer review

Post-publication review

This takes place whenever you read and appraise a journal article. All researchers and mental health professionals have a role to play in this, and their contribution is valued. Post-publication review simply describes the critique offered by readers of a published article, which presumably has previously been subject to peer review. Most of it occurs in isolation and is never communicated back to the authors. However, sometimes readers might publicise their opinions in the form of a letter to the editor, a blog article, a tweet or by contacting the corresponding author directly. This is to be encouraged, particularly from clinicians who might have unique experience and perspectives on the clinical or methodological area of research. Post-publication review regularly takes place in the journal clubs that constitute a component of training for doctors and medical students. Those who run such academic programmes should encourage attendees to write into journals where they feel that a paper presented has methodological problems deserving mention. For those who have never conducted formal peer review before, a constructive post-publication review can alert editors to a potential reviewer, prompting invitations to review for that journal.

Post-publication review sometimes results in a correction to the original article and is an important mechanism for identifying research fraud where this was missed by reviewers and editors (Godlee 2002; Haffar 2019; Wiley 2020).

Critics of the current peer review system have pointed out that, even when articles are found to have major flaws post-publication, some remain available to readers in the version originally published (Wiley 2020). It is therefore important that editors respond proactively to readers’ post-publication reviews as an essential means of ensuring the quality of available published research.

Transferable review

Transferable review refers to the process whereby reviews from one article are transferred to a different journal when a rejected article is transferred to that publication (Wiley 2020). Usually this occurs between journals belonging to the same publisher when an article is deemed more suitable for a lower-impact journal within the same family of journals. For example, the BJPsych editors sometimes offer transfer of a manuscript to BJPsych Open when an article is rejected by the BJPsych. Indeed, initial submissions to the BMJ involve selecting options from over 60 other BMJ journals (such as the Journal of Epidemiology and Community Health) that the author would consider transferral to if rejected by the BMJ (BMJ 2020).

The advantages of transferable review for both authors and editors are primarily in saving time, given the quite considerable work involved in reformatting a manuscript for a journal with another publisher (Wiley 2020). It also avoids duplication of work for reviewers. However, the main disadvantage of transferable review is that the time saving may persuade an author to concede transfer to a low-impact journal within the same publishing house at the cost of submitting it successfully to a higher-impact journal elsewhere.

Collaborative review

This can take two forms. In one approach, several reviewers work on a review together instead of submitting their individual independent reviews (Wiley 2020). This situation might arise when a senior researcher is asked to peer review a manuscript and informally passes it on to a junior colleague to review as part of their academic training. Once both have completed independent reviews, they meet to discuss their views, and the senior researcher submits a synthesis of the two reviews. In principle, such an approach should be agreed with the journal's managing editor, particularly as peer review comes with the expectation that the manuscript contents remain confidential.

As with all the other types of peer review, there are some advantages and disadvantages to this approach. It might lead to an enriched learning experience for new or junior reviewers and foster new collaborations. It might also lead to better quality reviews in synthesising the breadth of critique generated by a range of individual perspectives. On the other hand, a busy senior academic's overreliance on the reviews generated by junior researchers, without verifying the quality of their critical appraisal, may compromise their probity in accepting the review, particularly given the threats to confidentiality of the authors.

Another form of collaborative review is where authors and reviewers are encouraged to interact with each other through an online discussion forum while all concerns about the manuscript are addressed (Frontiers 2020). This is practised by Frontiers in Psychiatry and simulates repeated rounds of peer review but in a more interactive way. Additionally, authors are encouraged to feedback to Frontiers on their experiences of peer reviewers’ comments.

Hybrid forms

Hybrid forms of peer review also exist, whereby a manuscript might receive an initial single-blind peer review, but on publication of the article the names of the reviewers are published with it. This is currently practised by Frontiers in Psychology.

Evidencing peer review as continuing professional development

Clinicians and academics at all training stages are encouraged to engage in peer review as part of continuing professional development (CPD), and understanding the benefits and pitfalls inherent in the different forms of peer review described above is an essential foundation to their practice. Ultimately, our trust in the quality of published biomedical research rests on the individuals involved in peer review and the incentives that drive them.

Those who have little experience of reviewing research articles might initially feel daunted by the idea of participating in peer review, even if they feel confident in their critical appraisal skills. However, newer reviewers often have the most to gain (and offer) from the experience. Research shows that they tend to write reviews that are received as ‘less harsh’ or more constructive than those of senior reviewers (Casnici 2017). They are also quicker at returning their reviews (Casnici 2017), and there is some evidence to suggest that younger reviewers provide higher quality reviews (Goldbeck-Wood 1998; Halder 2011).

Agreeing to peer review a paper offers an opportunity to learn more about research processes and methods, as well as a preview of the most novel methods and research findings. Reference lists from articles can be useful for new researchers in the field, helping them read around a topic. Peer review may create opportunities for publication in the form of an invited editorial or commentary following a particularly thoughtful peer review. Research suggests that most early career researchers enjoy the experience of peer review, with 78% finding it to be positive (Rodríguez-Bravo 2017).

A 2015 survey showed that reviewers value an acknowledgement of the considerable amount of work they put into reviews (Warne 2016). The critical appraisal efforts involved in peer review can be evidenced in the case of open peer review, or in an email from a journal acknowledging receipt of a review. One way of recording the peer reviews one conducts is to register for a peer review registration site, such as that offered by Publons (publons.com). These sites offer greater recognition to reviewers by collecting evidence of all peer reviews completed for journals in one database. Metrics provided by these databases can be included in appraisal submissions and used as evidence of academic activity in the annual review of competence progression (ARCP) for trainees and in the annual CPD certificate for consultants.

There are other ways to reward high-quality peer review: International Political Sociology has started awarding prizes to outstanding reviewers (Lisle 2019). The BJPsych, which practises single-blind peer review, awards certificates of commendation annually to its top-ranking peer reviewers (ranked on quality and rapidity of review), and these count towards reviewers’ CPD. The EMBO Journal, which also practises single-blind review, has begun to publish reviewers’ comments anonymously so that others may learn from the process (Pulverer 2010).

Influence of review type on choice of journal when submitting

Given the above characteristics and relative benefits of the different types of peer review, authors should consider carefully where to submit their paper. As well as thinking about the remit of the journal, the quality of the paper in relation to the journal's impact factor and the average time taken to process a manuscript, an author should consider whether they would prefer open or blind peer review on the basis of the incentives and disincentives described above.

Whichever form of peer review is practised by the journal, submitting authors should ensure that they recommend as many potential peer reviewers as they feel able to, ensuring that none will have a conflict of interest. This helps editors by expanding the pool of peer reviewers and it enhances the chances that one of the reviewers contacted will agree to review. This is a particular problem for niche methodological or clinical areas, where the list of suitable reviewers might be short and the academic community relatively closed. Waiting months while the journal tries to find a willing reviewer is rarely acceptable to authors. Reviewers should check the status of their paper regularly, and where it seems to have been awaiting peer reviewer allocation for some time, they could email the handling editor to offer an expanded selection of peer reviewers.

How much is too much peer review?

When articles are rejected without review, a valuable opportunity for constructive feedback is lost (Pitman 2019). Even if a submitted article does not reach the quality threshold for publication in that journal, successive rounds of peer review and the evolution of improvements may be worth the efforts of reviewers and authors. The value of that input lies both in the learning and development of authors and in reducing the chances that original research findings go unreported. This process of peer review and editorial input might be regarded as a service provided to the academic community. However, research shows that the significant time cost to reviewers may not be acceptable to them in the context of their wider workload (Pitman 2019). Conversely, filtering articles more selectively could save editors valuable time, allowing them to focus on articles with a better chance of increasing their journal's impact factor and visibility.

The International Congress on Peer Review and Scientific Publication has recommended further research comparing the various forms of peer review to resolve many of the uncertainties described above (Haffar 2019), and it is encouraging that such studies are underway (Fox 2019). Until then, journals will continue to employ diverse peer review practices, basing their editorial decisions on an awareness of the disadvantages of each. In the age of digital information overload, readers rely heavily on the process of peer review in helping them decide which articles should influence their clinical practice (Smith 1997; Nicholas 2015). Readers, authors and peer reviewers will therefore benefit from an understanding of the biases and incentive structures inherent to each peer review process.

Box 1 offers tips on reviewing for potential reviewers.

BOX 1 Tips on reviewing and responding to review requests

• • Always be courteous in your review.

• • Remember that the person on the receiving end of your comments has put considerable time and effort into their article or review and may be at the start of their academic career.

• • Do not send a response when you are tired – if necessary, save your draft review, re-read it in the morning and then submit having gained this fresh perspective.

• • Help handling editors (and authors) process a manuscript more quickly by responding in a timely fashion to requests for review.

• • Decline reviews that are outside your area of expertise, and instead make suggestions to the handling editor as to who might be more suitable.

• • Decline reviews if you cannot commit sufficient time to the review within the deadline stipulated. Several hours are often necessary to do the review justice.

• • If you accept a review, try to respond within the time period suggested, and if you anticipate problems with this do contact the handling editor to discuss it.

• • Get into the habit of writing letters to journals to identify any serious methodological problems you identify in the papers that you read – this post-publication peer review forms a valuable contribution to the published literature.

• • Use peer review and your critique of others’ peer reviews as an opportunity to learn.

• • Record all reviews completed in your portfolio and/or on a peer review registration database.

Conclusions

The relative merits and disadvantages of the different approaches to peer review described here are important considerations when deciding whether to review a paper. These have implications for the quality of the review and, ultimately, the quality of a published paper. Although double-blind peer review has advantages in the reduction of specific biases, open peer review has the advantage of transparency. Self-awareness among reviewers of their own unconscious biases and any deficits in the methodological expertise required for a review will help improve the quality of peer review across the spectrum, enhancing the quality of published biomedical research.