Peer reviewing made easier: your questions answered

Summary

Peer reviewing is a hugely important part of the scientific process that ensures published articles are of sufficient quality to deserve dissemination to the wider scientific community. Building on a previous article published in this journal, this article addresses topics that potential or practising peer reviewers may find useful. These include what peer reviewing is, why do peer reviews, how to become a reviewer, what to write in a review and where to find more information. It includes a template for writing a review, and lists various websites and guidelines that can help ease the entire process depending on what type of article is being reviewed. Peer reviewing can be enormously rewarding and help clinicians diversify their scope of work while also benefiting the scientific community by contributing to the quality control of published work.

LEARNING OBJECTIVES

After reading this article you will be able to:

• • demonstrate a greater understanding of what the peer reviewing process entails

• • write a comprehensive review for a peer-reviewed medical journal

• • know where to go to for further information.

Peer review has been a formal part of scientific communication since the first scientific journals appeared more than 300 years ago. Philosophical Transactions, a journal launched in 1665, is thought to be the first journal to formalise the peer review process; it is also the first journal to be made freely available online (https://royalsocietypublishing.org/toc/rstl/1/1).

More recently, attempts have been made to establish peer reviewing guidelines. In 1978, a few editors met in Vancouver to establish guidelines for articles being submitted to their journals. This small group was named the Vancouver Group and continues to publish helpful advice and guidance in this area under the name of International Committee of Medical Journal Editors (http://www.icmje.org). This article uses some guidance from there as well as imparting knowledge and sharing learning from our own experience that should be useful for clinicians, academics, undergraduates and postgraduates in psychiatry.

Peer reviewing for medical journals is a richly rewarding experience benefiting the reviewer, author and journal. There appears to be a shortage of articles in psychiatry journals for those who have questions about the process and the details of peer review. This article aims to answer those questions and hopefully enthuse more people into becoming peer reviewers. It should be read in conjunction with our earlier article (Halder 2011), which discussed ‘the nuts and bolts’ of how to do it. Since 2011, N.H. has adapted and changed the style of peer reviewing based on experience and feedback, and this is shared in the present article.

What is peer reviewing?

A peer-reviewed journal is one that publishes articles that have been checked by people who are ideally experts in the field and is available to the wider scientific community. Peer review is the critical assessment by unbiased and independent experts of manuscripts submitted to journals. It can be seen as an important extension of the scientific process and it aims to validate academic work.

Peer reviewers are people deemed suitably qualified to act as independent assessors of such articles. They will have a duty to the authors to try to improve the standard of the article by constructive feedback. It can be seen as a form of ‘quality control’ for the journal. Indeed, 82% in a survey agreed with the statement that ‘without peer review there is no control in scientific communication’ (Ware 2016).

The reviewer will also have a duty to the journal to help the editor decide whether or not an article should be accepted (albeit with some amendments). Ultimately it is the editor who makes the decision to accept, revise or reject a paper.

The key stages of the peer review process are outlined in Box 1.

BOX 1 What is the process of peer reviewing?

1. 1 Editor receives a manuscript submission

2. 2 Editor can choose to reject straightaway

3. 3 If the manuscript has potential, editor will involve usually two peer reviewers for comments (picked from a database or suggested reviewers that the authors have provided)

4. 4 If there is a wide discrepancy in scores, an additional reviewer will be involved

5. 5 Editor makes final decision

Why do peer reviews?

The new decade has brought with it people in growing numbers who appear to be sceptical or even dismissive of the notion of evidence-based information, whether that is reflected in growing numbers of so-called antivaxxers (Hussain 2018) or climate-change deniers (Gross 2018) to name but two. It has therefore never felt more important a time to push back against this narrative and one part of doing that is either to engage in research or to act as a peer reviewer to ensure high quality publications.

There are a number of reasons to engage in the peer review process (Box 2). For one, it enhances critical appraisal skills. This helps to avoid losing the great skill that, for many, peaks at the time of their professional membership examinations. Some consultants mention a desire to engage in something different from their day-to-day routine. Not only does peer reviewing break up the clinical work, but it also adds a new skill set to consultants’ repertoire. It is something that can then be added to a CV and subsequently help in future job interviews. From a research perspective, reviewers benefit from reading about new research at the earliest stage and a chance to shape the final product. Simply having more knowledge about how and what reviewers look for increases the chances of having their own work published in future, as they will know what to look out for. As a general rule, it is desirable for peer reviewers to maintain an interest in a few specific areas of research where they have special knowledge.

BOX 2 Reasons for doing peer review

• • Enhances critical appraisal skills

• • Adds new skill set

• • Being part of the scientific research community

• • Good for CV

• • Breaks up clinical work

• • Helps with writing your own work for publication

• • Enjoyable

• • Contributes to the scientific process

• • Enhances reputation/career

Arguably most importantly, peer reviewing can be quite enjoyable. Part of the enjoyment could be the internal satisfaction of being part of an important process of quality control and making a contribution to the scientific process, ultimately benefiting their medical colleagues.

An added bonus, particularly for trainees and new consultants wishing to enhance their CV, is that many journals are now registered with Publons (publons.com), a free website that provides a service to track and verify the work of peer reviewers.

The most popular reasons for participating in peer review reported in one survey (Ware 2016) related to social factors (playing a part as a member of the community was endorsed by 93%, and reciprocating others’ reviewing work by 75%) and intrinsic factors (enjoy helping improve the paper: 83%; enjoy seeing work ahead of publication: 72%). Instrumental or self-interested reasons were much less cited (e.g. to increase the chance of future acceptances: 16%; to increase the chance of a place on the editorial board: 24%; to enhance reputation or further career: 42%).

Who should do it?

The answer to this is simple. Anyone can become a peer reviewer. All consultants can make a contribution, given the experience they have already gained in reaching this level in their career. However, trainees and even undergraduates can also make meaningful contributions. If anyone has completed a masters or PhD, it is likely that they will know more about the subject matter than many others. They can put themselves forward to be a peer reviewer for that subject. Trainees can certainly comment on a wide variety of issues, including education- and training-related subjects, having been ‘experts by experience’. A growing number of medical student journals are peer reviewed, when appropriate, by other medical students. This will help home in on skills from a very early stage, which will continue to improve with experience and time.

How to become a reviewer

You can contact the editor or the administrator of a journal in the first instance. All journals have email addresses or contact details on their website or inside the journal itself. As a minimum, the details that you should include in this first contact are your qualifications and experience, followed by your areas of expertise and interests (with details why you consider this to be so). If you have any previous publications these should be listed. You should also provide your CV. If accepted, your name and details will be placed on a database so that the editor can pick you if a submission that matches your expertise or interest comes in.

There are many hundreds of peer-reviewed medical journals available and it may be difficult to choose which one or ones to be a peer reviewer for. There are different ways to choose. For example, you can simply choose journals that you enjoy reading. Others choose by looking at affiliated bodies (e.g. only choosing journals that belong to the Royal College of Psychiatrists or another body). Others choose journals that tend to take submissions from a particular group, such as medical students or trainees. Some search by looking at the impact factor, which is considered to be a proxy measure for the quality of a journal (although not all peer-reviewed journals have an impact factor). Or you can simply list your areas of expertise and search on the internet for the journals that most closely match those areas. Readers are sometimes surprised at how accurately their interests are matched with a particular journal. As an anecdote, N.H., currently working with people with brain injuries, searched for journals on neurology, neurosurgery and psychiatry, as his work involves liaising with these groups of people. The first search item located was the Journal of Neurology, Neurosurgery and Psychiatry.

What to do if invited to review?

In some cases, you may be asked by a journal to review a paper after they found your details on a database such as PubMed or via a recommendation. (If, however, you have not reviewed a paper for a long time, you may be removed from the database of reviewers at the discretion of the editor.)

The Committee on Publication Ethics (COPE) has published helpful guidelines on things to look for before agreeing to take on a review (Committee on Publication Ethics 2020). These include checking that you have the necessary expertise to assess the manuscript. Note that if you would like to do the review but do not quite feel experienced enough, some journals would accept a review co-signed by your supervisor. Other important issues include declaring any conflicts or competing interests and only accepting an invitation if you can meet the deadline or have agreed an extension with the journal. Unfortunately, and sadly increasingly, you also need to check that the journal is legitimate. There has been an increase in so-called ‘predatory journals’, where certain open access journals exist for revenue rather than scholarly activity and often send out spam emails to solicit contributions with little concern for quality (Kisely 2019).

It is important to note that you do not have to ‘accept’ all requests for peer reviewing. Clinicians go through phases of being busier at certain times and there is no pressure to accept all requests for reviews that you receive. When this arises, you may be given the option to nominate or suggest other suitable reviewers for the paper. This is much appreciated by editors, who often find it difficult to get reviewers for highly specialised subjects.

How long does it take?

Of course, this is subjective and dependent on a number of variables. With more practice the time taken can decrease significantly. When N.H. first started peer reviewing as trainee editor of the Psychiatric Bulletin, the process could take a weekend of dipping in and out. Now it can take an average of about 2 h. This is in keeping with findings from a study in the British Journal of Psychiatry (Walsh 2000), which reported that the mean time taken to complete an open review (where reviewers’ names were published) was 2.05 h and 1.65 h for masked (‘blind’) reviews (where reviewers’ names were not known to the authors).† However, another survey (Ware 2016) found the median time per article spent reviewing was 5 h, with a reduction in the mean time needed per review with increasing age from the under-36s (9.6 h) to the over-65s (5.8 h).

You must resist the temptation of skim reading to save time. Authors have invested a significant amount of time from their busy schedule to complete the work that sits in front of you. It is crucial that you take enough time to give the authors the respect they deserve. You can, however, save time by only producing one review for both the authors and the editor, as these versions can be the same, with some very minor adjustments.

How do you approach a peer review?

Everyone may have their own way of approaching a review, and if it works for them then the advice would be to stick with this. Here is a suggested method. If you are not familiar with the journal, read the its aims and scope and instructions for authors to get an idea of what it is looking for. Once the full article is obtained, read it through once completely without breaks. Resist critiquing on the first read through, tempting though this may be. After short break, a second read through could be accompanied by written notes or questions that arise as you read the article. N.H. tends to print out the paper and write in the margins or between the lines. You may then write a first draft of the review (one way of setting this out is given in the next section). It is helpful then to leave it overnight or even longer. This allows you to organise your thoughts or formulate new ideas to help improve the article. Some say that if you go back to a review after a period it can sometimes appear that you are reading it with ‘fresh eyes’, generating fresh ideas. These can be incorporated into a revised draft. Finally, read through your review thoroughly to make sure it makes sense and does not itself have typos or grammatical errors. It is embarrassing to be critical of the authors’ poor grammar when your own writing is error-laden. For the review that the authors will see, you need to make sure that it addresses the main objective of improving the paper in a way that is encouraging and constructive. For the review for the editor, you have to ask whether this review helps the editor to make the decision about publishing the paper in their journal. Once you have completed and submitted your review you should destroy or delete any copies of the original article.

What do you write in a peer review?

Although a template was given in our earlier article (Halder 2011), N.H. has changed his approach to laying out a review, in light of experience and feedback. The original article advocates for a summary, followed by major points and then minor points. It is now felt more helpful for all parties to lay out the review in the order of how the authors have written their paper, using the subheadings that they use. This makes it easier to follow and appears to be logical. A summary is still useful to both editors and authors, but any recommendations about whether to reject or accept a paper should only be sent to the editor and not authors. Box 3 outlines a revised template for writing a review, and we discuss each element in more details in the rest of this section.

BOX 3 Template guide for structuring reviews

Give:

• • The article title

• • A summary of the review

• • Recommendation (to be included in review for editor only)

Next, address the following areas:

• • Title: Does this succinctly and accurately describe the main theme of the article?

• • Abstract: Does this summarise the main points from the text?

• • Introduction: Does the paper demonstrate an adequate understanding of the relevant literature and theory in the field?

• • Method: Can this be replicated on the basis of the available information?

• • Results: Are they accurately described using the correct empirical tools?

• • Discussion: Do the conclusions follow clearly from the arguments/points made?

• • Limitations: Does the paper recognise its limitations?

• • References: Are they relevant, up to date and correctly cited? Has any important work been omitted?

Finally, give a statement under each of the following headings:

• • Originality

• • Readability

• • Topicality

• • Validity

• • Likely appeal

• • Generalisability/implications

Title

In practice this means that (after the brief summary) you should start with the title. Does it accurately reflect the work conducted? Some journals specifically state that they do not want the title in the form of a question. Some journals have guidelines on the word count and format of the title, but that can be left to the copy editors. The reviewers can suggest amendments to the title or even alternatives as they see fit. Some journals will provide a specific template for reviewers.

Abstract

The next section is usually the abstract. Together with the title, these make up the first thing readers will read when they are searching for journal articles. If the article is not open access (meaning freely available on any platform in full) then the title and abstract could be the only text that is visible without having to pay, subscribe or have special authorisation to the database in which the article is archived. The abstract should therefore not be overlooked or only given passing remarks. The abstract should contain the key points and findings from the main text succinctly. The key question to answer is: Does this accurately reflect the main text? Previous work has highlighted this can be a major problem (Pitkin 1999).

The rest of the review should follow the subheadings as given in the article being reviewed; for the majority of scientific papers this will follow the IMRAD headings (Introduction, Method, Results and Discussion).

Introduction

A reviewer after reading the Introduction section should be able to answer the question: Why was this paper written? The authors should provide a brief context or background for the study (rather than a whole history on the topic). Any purpose, aim, objective or hypothesis should, ideally, be included here. Sometimes authors include data or conclusions from the work being reported, but these should appear later in the relevant section. Comments should be made about the theoretical underpinnings to the work carried out. Does the paper demonstrate an adequate understanding of the relevant literature and theory in the field? Is this referenced appropriately? Is the theory clearly and succinctly presented within the submission? Is any significant work ignored? Ideally, as the reviewer you should already be an expert in the field and so should know the relevant papers and omissions. If these are not at your fingertips, it is worthwhile doing a literature search of your own to check.

Method

Regarding the Method section, the questions to ask include: are the aims of the paper clearly stated? Is the methodology described sufficiently so that it could be replicated? Are the correct statistics used? If it is a primary research paper, are there power calculations? Authors should clarify why any variables or cut-off points were used. If ethical approval was necessary, there should be information contained here. You should comment on any omission of this. Likewise, concerns of misconduct should be sent to the editor immediately.

Results

Results should be presented clearly. You can note if you feel there is a better way of conveying the results that the authors have overlooked. Tables and figures are most often included in this section and the editor expects the peer reviewers to comment on these. Specifically: Do the tables capture information concisely and display it efficiently? Do they provide enough information? Including data in tables rather than text frequently makes it possible to reduce the length of the text. Comment if data in tables or figures are repeated in the text. As a reviewer you may be confident enough to check the statistics yourself, but if in doubt you can ask the journal statistician to comment. If there is no in-house journal statistician, you need to be open and honest about your remit and say this in the review. How adequate is the analysis, including qualitative and/or quantitative analysis?

Discussion

The Discussion section should adequately tie together the other elements of the paper. It should answer the question: What do the results mean? Do the conclusions reached follow clearly from the arguments/points made? Have the authors emphasised any new and important aspects of the study? As a peer reviewer it is important to check that authors have not repeated in detail data or other material given in other sections. The conclusions should be linked with the aims of the study, and any unqualified statements and conclusions not adequately supported by the data should not be there. A paragraph on limitations of the study is useful here. As a peer reviewer you need to ask: Does the paper recognise its limitations and consider these well?

References

With references, you would have already mentioned any omissions. It is more helpful to provide details and specifics of the references you think are important instead of using airy language such as ‘the authors should be aware of the work by Bloggs et al carried out in the 1990s’. Include some comments about whether you think the references provided are relevant, up to date and correctly cited. This includes checking that authors have adhered to the journal style. Copy editors and typesetters are invaluable here in checking the article is in keeping with the journal requirements, but many are paid per page or per 1000 words so it can become costly to fix references and other areas that are not to style. References to papers accepted but not yet published should be highlighted; authors should obtain written permission to cite such papers, as well as verification that they have been accepted for publication.

Other areas for comment

After commenting on the sections above, it is useful to write a paragraph on the following areas.

Originality

Original ideas make for interesting reads, and medical journals, like newspapers, need readers to survive.

Readability

This would include whether there is a logical flow throughout. Each section should flow from the one preceding it. The aims described early on should be addressed in full. Some journals have guidance on how the paper should read. For example, some favour short, succinct sentences and ask authors to avoid the passive sentence structure. You can include comments on the English style and grammar. It is also acceptable for reviewers to simply comment that the spelling and grammar are poor, rather than going through with a fine-tooth comb to correct it themselves, and there is ongoing debate whether this task falls within the remit of reviewers at all (see ‘Controversies in peer review’ below).

Topicality

This links in with originality in the sense that readers are more likely to read articles on topics that are trending or being actively discussed in scientific or wider circles for whatever reason.

Validity

Have there been any other studies that can confirm the authors’ findings? If the study is the first of its kind, there should be a mention of having to replicate it to ascertain that the results have not been due to chance alone.

Likely appeal

This relates to the appeal for that particular journal and the appeal to its readership. Many papers are rejected because the question being asked is not interesting enough or does not add enough to the body of scientific evidence.

Generalisability or implications for practice, research and/or society

Does this paper inform practice for the reader and contribute to the field of theory, research and/or practice?

Are clear implications for the reader demonstrated? Are the implications consistent with the findings and conclusions of the paper?

Throughout, the reviewer should always have in mind what can be done to improve the paper, and this question alone should guide all the writing.

How to write a review when the paper is seriously substandard

Because the peer review process includes confidential comments to the editor that will never be seen by the authors (unless they go so far as to request information via the Freedom of Information Act) the frank opinion of a very poor paper should be made primarily to the editor. The comments to the authors should be robust, accurate and relevant, but not rude. The first paper one of us (P.T.) sent many years ago to the British Journal of Psychiatry received an angry response, ‘Why on earth is this registrar carrying out research at his age when he should be carrying out his clinical duties?’ This now would not be tolerated.

A very poor paper will naturally tend to receive a shorter report than a better one.

Controversies in peer review

Journals that do not operate a peer-review process tend to be considered of not high quality because of the risk of editorial bias and, at worst, fraud. A survey revealed that, in addition to detecting the best manuscript for the journal, growing numbers of people believe that peer reviewers should also be looking to detect fraud (i.e. the results given are falsified) and plagiarism, although acknowledging difficulties in actually doing this (Ware 2016). Indeed, one study found that, out of 400 consecutively submitted manuscripts to a major specialty medical journal, 17% of submissions contained unacceptable levels of plagiarised material, with 82% of plagiarised manuscripts submitted from countries where English was not an official language (Higgins 2016). However, to detect fraud, reviewers would potentially need to check and verify the raw data of a study, which could be impractical and discourage prospective reviewers from participating in the peer review process (Glonti 2019). Some may argue that peer review should not involve ‘policing’ other people's work, especially when there is sophisticated software available, such as Turnitin (turinitin.com), that can be tasked to do this. The Royal College of Psychiatrists has recently formed a Research Integrity Group and this will be focused on, among other things, the role of reviewers in spotting fraudulent, plagiarised or poor research.

There is an ongoing debate whether peer reviewers should engage in copy-editing. Although the majority of articles examined stated that copy-editing does not fall within the duty of peer reviewers, several articles specifically mentioned that reviewers should offer grammatical and linguistic improvements (Glonti 2019). Many journals have their own copy-editing team, who are specifically trained to identify and address such aspects of the manuscript. Reviewers may argue that their time should be focused on the improvement of scientific content rather than the linguistic fine-tuning.

Where else can I find help as a reviewer?

There are guidelines freely available online for almost all types of research (Box 4). These can be consulted to help inform your review, and they are especially useful if you are struggling to benchmark what has been written against a standard. The International Committee of Medical Journal Editors (ICMJE) provides guidelines for biomedical journals in general (http://www.icmje.org).

BOX 4 Guidelines for particular studies that may help reviewers

• • Observational studies: STROBE

• • Systematic reviews: PRISMA

• • Study protocols: SPIRIT

• • Diagnostic/prognostic studies: STARD

• • Case reports: CARE

• • Clinical practice guidelines: AGREE

• • Qualitative research: SRQR

• • Animal pre-clinical studies: ARRIVE

• • Quality improvement studies: SQUIRE

• • Economic evaluations: CHEERS

Links to the above guidelines can be found at https://www.equator-network.org

COPE produces guidelines on the peer reviewing process from an ethical standpoint, including what to do if you suspect fraudulent practices (https://publicationethics.org/peerreview).

The Royal College of Psychiatrists runs annual or biannual 1-day research methods courses. There are hands-on and practical, facilitated by experienced reviewers. These are advertised on the College website and in the newsletter.

Conclusions

We hope that this article, alongside our original article (Halder 2011), has enthused a new generation of reviewers and continues to be a source of guidance for existing reviewers. Peer reviewers play a crucial role in the editorial process, and yet their role and tasks tend to be poorly defined (Glonti 2019). We hope this article has helped answer some of the key questions involved in this process. N.H. is happy to be contacted if needed.