Peer reviewing made easy

Summary

Peer review in scientific journals has existed for over 200 years. This process is currently the accepted way of assessing scientific manuscripts prior to publication for most journals. Despite this, little has been written about the process of peer reviewing, with hardly anything specific to psychiatric journals. This article answers fundamental questions related to peer reviewing and includes practical tips to writing reviews. It will be helpful for those keen on improving their knowledge about the peer-review process. It will not only benefit those who are reviewers or thinking about becoming reviewers, but also authors, who can use the information to improve their chances of publication.

Every author can relate to the satisfaction of completing an article or piece of research. There remains only one thing in the way between submission and publication – the peer review. Will all that hard work and investment of time pay off? The wait may be agonising for the seasoned professional as well as for the novice.

Before accepting the role of Trainee Editor for the Psychiatric Bulletin (now The Psychiatrist) in 2008, N.H. had not peer-reviewed for a journal. The literature on this subject was sparse, with no clear guidelines on how to write a review. At a workshop that we ran at the Royal College of Psychiatrists’ Annual Meeting in 2009, one of us (N.H.) described the principles of how to carry out peer review. This generated considerable interest from many colleagues desperate for information on the topic. This article is the result of amalgamating information from a variety of sources, including feedback from the workshop, discussions with senior peer reviewers and editors of the College's journals, and consulting papers and books written on the subject.

What is peer review?

Peer review is a ‘formal system whereby a piece of academic work is scrutinised by people [the peer reviewers] who are not involved in its creation but are considered knowledgeable about the subject’ (Wager 2002).

The peer-review process is important and all reviewers have a duty to ensure that the highest possible standards are met. The quality of this process is rightly cited as one of the main influences on a journal's reputation and standing (Scott-Lichter 2009). It serves as a ‘quality control’ mark for published articles.

Peer reviewers may find it helpful to think of themselves as ‘consultant advisors’. The reviewer has two broad aims, which are directed at two main parties:

1. • the author(s) – to help improve quality and therefore the chances of publication (in any journal)

2. • the editor – to aid decision-making.

These should be at the forefront of any reviewer's mind. Ultimately, the editor makes the decision. It would be wise to be mindful of not conveying your decision to the author as this could be construed as going over the editor's head. More information about the reviewers’ responsibilities towards authors, editors and readers is available from the Council of Science Editors’ White Paper on promoting integrity in scientific journal publications (Scott-Lichter 2009).

Types of review

Broadly speaking, there are two types of review – open and masked reviews.

Masked reviews

Masked means the same as ‘blinded’ review. As in randomised controlled trials, such concealment can be either single-blind, where the authors do not know the identity of the peer reviewers, or double-blind, where neither authors nor reviewers know each other's identities.

Advantages

Reviewers may be prejudiced by an author's identity or place of work; masking theoretically reduces personal bias and makes the review more objective. Some editors believe it improves the quality of reviews (Yankauer 1982; Robin 1987). It appears that most reviewers favour anonymity from authors. In one of the largest international surveys of authors and reviewers, 58% of the 4037 researchers said that they would be less likely to review if their signed report was published and 76% favoured the double-blind system where only the editor knows who the reviewers and authors are (Sense about Science 2009).

Disadvantages

It is difficult and expensive to keep the author's identity hidden from a reviewer. This is because authors tend to cite their own work and, in certain fields, researchers are familiar with one another's work. One study showed that up to 42% of reviewers who were not told the identity of the authors were still able to identify them (Goldbeck-Wood 1998). Another problem is that reviewers may run the risk of abusing their position because they do not feel accountable, a point highlighted by the Council of Science Editors (Scott-Lichter 2009).

Open reviews

Open reviews are when reviewers and authors know each other's identity. Reviewers are asked to sign their own work and some have their comments posted on the journal's website.

Advantages

It makes the process transparent and there is greater accountability of reviewers. Fiona Godlee, Editor of the BMJ, argues that open review is ethically superior for the reasons given above (Godlee 2002). She agreed with other authors (Fabiato 1994) that open reviews would decrease opportunities for making judgements that were unjustified or biased because reviewers could not hide under the ‘cloak of anonymity’. Reviewers may feel the fruits of their hard labour go unnoticed, but open reviews mean they can get credit for their work.

Disadvantages

Reviewers may feel inhibited about expressing their true feelings. This is especially apparent when junior reviewers are asked to review the work of senior colleagues, and it might lead to more reviewers declining work. There is always the possibility of bullying tactics to place pressure on junior reviewers to accept an article for publication. Despite these disadvantages, the BMJ adopts an open peer-review system, but reviewers can express their concerns anonymously if they feel intimidated (Smith 1999).

Advantages and disadvantages of a peer-reviewed journal

A peer-reviewed journal is one in which submitted articles are independently examined by a panel of external experts prior to publication. The Psychiatrist and the British Journal of Psychiatry have at least two independent peer reviews before approving an article for publication, and Advances in Psychiatric Treatment usually uses one reviewer.

Advantages of peer review

1. • Peer reviewing normally helps ensure the journal maintains a certain standard.

2. • It is more likely that errors and flaws in the paper are detected before it is accepted for publication, which helps to weed out unsatisfactory papers.

3. • It helps to separate original papers from ‘me-too’ ones and prevents potential biases, particularly those of the editor and editorial board.

4. • It acts as quality assurance. In the Peer Review Survey (Sense about Science 2009), 91% of authors said that their latest paper had been improved through peer review, the discussion section being the biggest area of improvement.

5. • Peer-reviewed journals are held in greater esteem than non-peer-reviewed journals.

Disadvantages of peer review

1. • It adds a considerable delay between submission and publishing (e.g. for The Psychiatrist it is about 8 months, although this time is getting shorter).

2. • Reviewers may know or guess the identity of the author.

3. • Revolutionary or unpopular conclusions drawn by the authors may be rejected by reviewers (e.g. portraying psychiatrists in a bad light).

4. • There is potential for personal bias (i.e. depends on what the peer reviewers like and dislike).

5. • Publication bias still operates (i.e. papers reporting positive findings are more likely to be accepted).

6. • Papers may be turned down because of rivalry between the reviewer and author, irrespective of scientific merit.

The role of an editor

The editor is the person who has ultimate responsibility for the direction and content of the journal. When a scientific paper is submitted, the editor has to read the paper to ascertain that it is deserving of peer review. Sometimes the initial in-house review is carried out by a member of the editorial board with expertise in that particular area. Many journals have a committee to decide on this – in the British Journal of Psychiatry it is known as the Janitor Committee.

After review, it is the editor who makes the final decision on whether a paper should be published or not. The role of the reviewer is to act as advisor to the editor and their recommendations may be overridden in certain circumstances, such as when the editor believes the reviewer did not adequately consider the paper, made injudicious comments or made a recommendation to publish without appreciating the pressures on space in the journal. In circumstances where the editor is dissatisfied with the standard of the review, a further review may be sought. Some journals, such as the BMJ, have a ‘hanging committee’ whose role is to aid the editor in reaching a decision on publication in the event that there is uncertainty.

When two reviewers disagree on one paper, the response of the editor will differ depending on the stage of the review process. If it occurs early in the process, for example after the initial submission, it is important to read both reviewers’ comments carefully since the score given may not be commensurate with their review or it may be apparent that the reviewer has not been diligent enough or has been excessively harsh. The editor can then make a decision based on their own evaluation of the merits of the paper, taking into account the reviewers’ comments. Alternatively, another reviewer may be selected to replace the outlier. If the disagreement occurs after the paper has been resubmitted, the editor may make the decision with an explanatory letter to the author or ask for a third review. In this way the likelihood of an appeal is lessened and may be seen as fairer by the author.

The editor of a journal has many different roles and is extremely busy. Good reviewers help enormously in aiding the decision-making process.

How an editor chooses reviewers

Submitting papers to journals is now mainly done electronically. The process is usually explained on the journal's front or back cover, or on its website. Softwares may differ between journals but essentially, when authors submit papers, their names are added to the e-submission system as potential reviewers in their field, whether their papers are published or not. Those who have expressed an interest in reviewing will also be added, thus building up a list of reviewers. All are asked to provide information on their areas of expertise.

Performance as a reviewer is monitored automatically to include the number of reviews carried out over a set period, the turnaround time and dates of the reviews. There is a facility in some databases for the editor to rate the quality of the person's reviews. However, these ratings are not shared with the reviewers. A drawback is that the database needs regular updating so that it is amended when a reviewer moves or develops another area of expertise, and it is the reviewer who is responsible for this. The lists on many databases are out of date.

If a reviewer is contacted but does not reply within a specified deadline, the reviewing software alerts the editorial office. The reviewer invitation process then begins again. Thus, failure to update the database has the effect of increasing the time to publication for the individual paper. P.C. (Editor of The Psychiatrist) once had to contact 14 possible reviewers before finding one who would agree.

What to do on receipt of invitation to review

It is a privilege to peer review a paper. The reviewer plays an important part in the advancement of scientific knowledge and ensuring that the quality and standards of the journal are met. Being a peer reviewer is a great addition to a CV: for trainees it covers several of the competencies required by curricula for the portfolio; for specialty doctors and consultants it will help with revalidation. This does not mean one should merely jump at the chance when asked to review, tempting as it may be to agree with H.G. Wells that ‘No passion in the world is equal to the passion to alter someone else's draft’. The role is one that should be taken responsibly and seriously. Wager et al(2002) suggest that one should ask oneself five preliminary questions before accepting a manuscript for review (Box 1). In the rest of this section we expand on their discussion of these questions from the point of view of an invited reviewer.

BOX 1 Questions to ask when invited to review

1. 1 Is the manuscript within my field of expertise?

2. 2 Am I happy with the journal's peer-review process?

3. 3 Do I have time to do this review?

4. 4 Can I meet the deadline?

5. 5 Do I have any conflicts of interest?

(Wager 2002, with permission)

Is the manuscript within my field of expertise?

Ideally, you need the manuscript to be on a field you know well and you should have good knowledge of the current literature. If you have inadequate knowledge about the content or methods and feel unable to write a good review, it is alright to decline the invitation. If you feel able to write the review but bits of the manuscript are outside your field of expertise, it is important to declare that to the editor. The editor may choose to consult additional reviewers. When the audience at the College Annual Meeting was asked what was the most commonly encountered problem with regard to this scenario, the overwhelming response was ‘statistics’. You can ask for independent scrutiny from a statistician. Some editorial boards have specialist statistical advisors (the British Journal of Psychiatry) and others do not (The Psychiatrist).

Am I happy with the journal's peer-review process?

The three main College journals (the British Journal of Psychiatry, The Psychiatrist, Advances in Psychiatric Treatment) operate a single-blind review system (see above). If you feel uncomfortable with the review system, you should raise your concerns with the editor. If your concerns cannot be addressed, you may have to decline the offer to review for that journal.

Do I have time to do this review?

As a novice reviewer (certainly compared with the other authors of this article), N.H. takes about 4 h, with a range of 2–5 h, to review papers submitted to The Psychiatrist. Some suggest that first-time reviewers should put aside 8–12 h, with some complex submissions taking up to 48 h (Wager 2002). Others have found that, on average, reviewers spend 2–4 h on a review (Yankar 1990). One study looking at the quality of reviews for articles submitted to the BMJ (as rated by editors and authors) found that quality increased with time spent on a review, but only up to 3 h and not beyond (Black 1998).

Can I meet the deadline?

Most journals ask reviewers to complete a review within a few weeks. The British Journal of Psychiatry and The Psychiatrist request a turnaround time of 3–4 weeks. Some software databases for reviewers have an average review time that is tabulated and can be seen by the editor. It is frustrating for an author to wait for a decision on a paper, so you should agree only if you can deliver on time.

Do I have any conflicts of interest?

Conflicts of interest include anything that might unfairly affect the reviewer's view of the manuscript (either positively or negatively). The most common is working with one of the authors (either currently or previously). If you are referred to in the text of the submission, it is important (but not easy) not to let that bias your judgement. There may be other conflicts of interest, such as financial (e.g. having shares in a company), political or religious. For example, the Psychiatric Bulletin published an article on psychiatry and religion, which may not have been suitable for review by someone whose strong beliefs might have prejudiced the outcome (Koenig 2008).

The most common reasons reviewers give declining to review a manuscript are lack of time and the paper not being relevant to their area of interest or expertise (Tite 2007).

What to do on receipt of the paper

We recommend the following first steps on receipt of a manuscript that you have agreed to review.

1. • Right from the start, ask yourself whether the paper suits the journal. If you are not familiar with the journal, the editorial office can send a copy of the instructions to authors. This will act as a guide as to whether the authors have successfully adhered to what the journal is looking for. You may wish to read other articles published in the journal to get a feel for the style and level of what gets published. It may help to see whether the reference list in the submitted paper contains papers published in the same journal.

2. • Read through the paper once fully, resisting the temptation to start critiquing it at this stage.

3. • Make a note of first impressions. Unlike reviewers, most readers will only read it once. Therefore your first impressions are likely to be similar to those of other like-minded colleagues. If it does not make much sense to you, it probably would not make sense to other readers.

4. • Read the paper again slowly. This time make notes. Guidance to what is required is shown in the template in Fig. 1.

   

   FIG 1 Template for peer review.

5. • You may find it helpful to then sleep on it and come back to it the next day. This may give new insights on previously missed points.

6. • Bear in mind that two reports need to be written (one to the editor and one to the author), but often the same report can go to both author and editor.

7. • In the case of research papers, you may find it helpful to use the PICO rule (Box 2) as guidance to make sure that the basic elements are covered.

8. • Some reviewers prefer an even simpler checklist – the four ‘W's (Box 3) (Albert 2009). You should be able to adequately describe to a colleague the points covered by the four items.

9. • The underlying factor driving your critique should be the scientific merit of the paper.

BOX 2 The PICO rule

1. • Patient or Problem – is it well-defined?

2. • Intervention – what is the main intervention, exposure, test or prognostic factor?

3. • Comparison – is there a comparative intervention? (not always needed)

4. • Outcomes – did the authors describe what they hope to achieve, measure or affect?

For example

In people who have relapsed after at least one previous episode of schizophrenia (P), what is the likelihood of olanzapine (I) compared with haloperidol (C) producing better reduction in positive symptoms (O)?

(After Gosall 2009, with permission)

BOX 3 The four ‘W's

1. • What the authors looked at (usually covered in the introduction)

2. • What they did (the method)

3. • What they found (the results)

4. • What this means (usually in the discussion and is the key point)

(Albert 2009)

How to write the review

Templates can be useful for making sure nothing major is omitted. This is rather akin to many trainees who learn mental state examinations by initially using templates to remember all the sections. However, even for more experienced reviewers, templates such as that shown in Fig. 1 can be useful.

The summary of the review ideally should contain no more than three sentences. The first should describe the topic and method. The second should comment on major shortcomings and/or changes that need to be made. The third should make helpful concluding comments to the author/editor about your overall opinion. Following this, the review can be broadly split into major and minor points (Fig. 1).

An alternative is to go through the paper systematically in the order the authors have written it, writing a paragraph under each subheading. These paragraphs should be numbered to make it easier for authors to respond. For example, starting with the abstract, this is often the first and only thing many psychiatrists read when searching the literature. Bearing this in mind, you need to make sure that it accurately reflects the body of the manuscript. Note that before accepting or declining the review, you will have been able to read the author's abstract.

You then need to check that there is a stated aim or hypothesis. Following this, it is important to make sure the method is clearly described, valid and appropriate to the question posed. The results should not be repeated in the discussion and the conclusions should be based on the results. The conclusions should be true, accurate and justifiable from the data available. The discussion should be appropriately focused. You might suggest additional references, which can be helpful to authors. It is important to scrutinise the references – if they are too old they may be inappropriate. It is pertinent at least to read the abstracts of the references listed since authors sometimes cite inappropriate references without having read them.

Editors rightly place a lot of emphasis on grammar, syntax and language. For example, Advances in Psychiatric Treatment aims for concise, lucid and unambiguous language (Bouch 2009). Although journals may have copy-editors who check such detail, it is good practice for reviewers to pick up any errors and comment in the specific section of the review.

You may need to do your own literature search and add it to the review. This is important when discussing originality, if only old papers are referenced or if one is dealing with a paper concerned with peripheral or esoteric areas (e.g. the reviewer has an interest in suicide and is asked to review a paper on suicide by motorcycle accident).

Remember, no paper is perfect and most papers require revision.

The length of review will depend on the complexity of the individual manuscript and how much you have to say. One- or two-lined reviews are not acceptable; nor is raising new methodological flaws on a redrafted paper that were not spotted on a previous review. The words of Einstein ring true here: ‘Everything should be made as simple as possible, but not simpler’ (Calaprice 2005).

A summary of the do's and don'ts of writing a review is presented in Box 4.

BOX 4 Do's and Don'ts of peer reviewing

Do:

1. • make sure to be courteous

2. • point out the positive

3. • ask for a statistical review if you are uncertain

4. • structure your review

5. • divide the review into major and minor concerns

6. • be objective

7. • whenever possible offer evidence for your views and opinions

Don't:

1. • personalise the review

2. • make intemperate comments

3. • assume that it's your fault if you don't understand a paper – it may be unintelligible

4. • tell the author what study you would have done unless methods are flawed

5. • write one-liners (e.g. ‘this is a poor paper and should not be published’)

6. • raise new methodological flaws in a redrafted paper that you didn't spot in a previous review. It is unfair to authors who have acted on all previous recommendations.

What to do after the review has been completed

For the British Journal of Psychiatry and The Psychiatrist reviewers are asked to score the submission on a 5-point rating scale (Box 5), but other journals will have a different scoring system.

BOX 5 Scoring system for reviews

1. 1 Excellent

2. 2 Strongly recommended (minor changes)

3. 3 Potential value (significant changes)

4. 4 Not for acceptance (but may be rewritten as a letter?)

5. 5 Rejected

Very few manuscripts are accepted first time (P.T., Editor of the British Journal of Psychiatry, estimates less than 1%). Joe Bouch (2009) stated that the record during his first 2 years as Editor of Advances in Psychiatric Treatment was six resubmissions. For the Psychiatric Bulletin, out of 214 new submissions between October 2007 to September 2008, 21 (10%) were rejected without review, 32 (15%) were rejected but invited to submit the paper as a letter, and 40 (19%) were rejected after review. The total number rejected was 93 (44%) and the total number accepted following review was 108 (50%). The remaining 13 (6%) were withdrawn by the authors or deleted by the editors.

Once you have submitted your review, you have to wait for the editor's decision, which is usually communicated via email several weeks later.

What makes a good reviewer?

Some authors believe a good reviewer is one that combines a critical eye with a positive creative attitude aimed at improving manuscripts and educating fellow researchers. The best reviewers concentrate on offering useful advice to authors rather than giving summary judgements to editors. Other core attributes include courtesy, punctuality, thoroughness, keenness and being knowledgeable in the discipline (Golbeck-Wood 1998).

Many younger reviewers of junior academic status are reluctant to review, feeling inadequately skilled, especially when the author is a well-known senior figure. However, the evidence suggests that younger reviewers write better-quality reviews (Stossel 1985; Evans 1993; Kliewer 2005). Research training and postgraduate qualifications are not associated with review quality. Black et al(1998) attempted to determine the characteristics of reviewers who produce high-quality reviews for the BMJ as determined by both editors and authors. In a logistic regression analysis, the only significant characteristics were younger age and those with training in epidemiology or statistics. Other authors have found no significant correlation between quality of reviews and gender, years of reviewing and academic rank (Kliewer 2005).

Obtaining feedback is always a good thing in order to support reviewer development. Many journals send reviewers the comments of other reviewers of the same paper. You can also ask the editor for feedback, perhaps using a tool developed by Van Rooyen and colleagues (1999) called the Review Quality Instrument (RQI). This instrument considers seven aspects of a review (importance of research question, originality, methodology, presentation, constructiveness of comments, substantiation of comments, interpretation of results), each scored on a 5-point Likert scale (1, not at all or poor; 5, discussed extensively or excellent). The total score is the mean of the 7-item scores. There is an additional eighth item for the overall assessment of the quality of the review. One may assume that reviewers who work through questions from the RQI (Box 6) might write better reviews.

BOX 6 Questions from the Review Quality Instrument (RQI)

1. 1 Did the reviewer discuss the importance of the research question?

2. 2 Did the reviewer discuss the originality of the paper?

3. 3 Did the reviewer clearly identify the strengths and weaknesses of the method (study design, data collection and data analysis)?

4. 4 Did the reviewer make specific useful comments on the writing, organisation, tables and figures of the manuscript?

5. 5 Were the reviewer's comments constructive?

6. 6 Did the reviewer supply appropriate evidence using examples from the paper to substantiate their comments?

7. 7 Did the reviewer comment on the authors’ interpretation of the results?

8. 8 How would you rate the quality of the review overall?

(After Schroter 2006, with permission of the American Medical Association)

The bottom line is that editors love to hear from enthusiastic potential reviewers of any age and background who feel that they can do a good job. After reading this article, we hope that you are in a better position to do just that.

Reviewing books

Writing a book review is a good way of getting your own name in print, perhaps for your CV. You will be sent a book or books to read and asked to write a review. It is tempting simply to list the contents and contributors and to make a comment on the potential audience, length or cost of the book. More interesting for the reader, however, and a challenge for the reviewer is to write a short article using the book as a starting point.

Writing to potential reviewers, the book review editors of the British Journal of Psychiatry and The Psychiatrist comment that a review is an opportunity to inform the reader of a book's merits, but should also be interesting in its own right. Pulling out one or two themes and exploring how the book has been approached requires careful reading and thinking about what the book is presenting.

You can develop your skills as a book reviewer by taking on different types of book to review, from single-author titles focused on a specific issue to books covering more varied material and with sometimes many contributors. To quote Simon Wessely, ‘a review should inform, entertain and occasionally provoke’ and the best reviews ‘are often jewels of argument and exposition, essays in their own right’ (Wessely 2000). You should respect the amount of time that goes into writing and editing a book; this is not to say that a review cannot be critical – but any critical comments should not be offensive (Crown 2000).

To find out more

The following books and articles may be useful to peer reviewers of today and tomorrow.

1. • The Doctor's Guide to Critical Appraisal (Gosall 2009)

2. • How to Read a Paper (Greenhalgh 2000)

3. • How to Write a Paper (Hall 2008)

4. • Critical Appraisal for Psychiatrists (Lawrie 2000)

5. • How to get Published. All you Need to Know to Publish in Psychiatry (Baylé 2006)

6. • How to Survive Peer Review (Wager 2002)

7. • ‘A systematic guide to reviewing a manuscript’ (Provenzale 2005).

A great deal of emphasis is currently placed on psychiatrists’ ability to systematically appraise published literature, and it now forms a key part in the MRCPsych exams and journal clubs. Interestingly, there is little such emphasis for adopting a robust and systematic approach to peer reviewing submitted manuscripts.

Next steps: how do I become a reviewer?

If you wish to become a reviewer, you should contact the editor of the journal and explain why you would like to do this work. You should also indicate your areas of scientific interest. It is important to provide a reference attesting to your ability to critically consider papers and also to meet deadlines. You will be required to demonstrate that you have a true expertise in a particular area, not simply a more-than-average interest, since you are likely to be asked to consider papers by authors who have published extensively in a particular area.

Good reviewers are highly sought after and desperately needed. In the Peer Review Survey (Sense about Science 2009) one in five researchers questioned thought that peer review is unsustainable owing to few willing volunteers. But 86% of volunteers enjoyed peer reviewing, although the majority said that there is a lack of guidance and that formal training would help. This article goes some way to addressing the issue.

MCQs

Select the single best option for each question stem

1. 1 Among the advantages of masked review over open review is that:

   1. a it is cheaper and easier

   2. b it reduces personal bias

   3. c it offers more transparency

   4. d it makes the reviewer more accountable

   5. e authors can communicate with reviewers.

2. 2 An advantages of a peer-reviewed journal is that:

   1. a reviewers can guess the identity of the author

   2. b there is no chance of personal bias operating

   3. c peer review helps to weed out unsatisfactory papers

   4. d positive findings are more likely to be accepted

   5. e there is less delay from submission to publication.

3. 3 The role of a peer reviewer is to:

   1. a decide whether a paper should be published in a journal

   2. b rewrite the paper to correct flaws

   3. c help the editor in the decision-making process

   4. d write only to the editor regarding a decision

   5. e criticise the author.

4. 4 A peer review should include:

   1. a personal comments to the author

   2. b a score out of 5 to the author

   3. c comments on the generalisability of the study

   4. d comments on how you would rewrite the methods

   5. e comments on your own research in the field.

5. 5 Factors that have been shown to positively influence the quality of a reviewer's work include:

   1. a being of a particular ethnic origin

   2. b the seniority of the reviewer's position

   3. c having a research-based degree

   4. d punctuality in writing reviews

   5. e being female.

MCQ answers

1

b

2

c

3

c

4

c

5

d