Notes from the Editors: Post-Publication Critiques

One important purpose of peer review is to ensure that work published in scholarly journals is as accurate as it can be. The American Political Science Review is fortunate to have many outstanding reviewers who check submitted articles carefully, ensuring not only that the work makes a worthwhile theoretical or substantive contribution to ongoing scholarly conversations but also that the empirical pieces are reliable. Reviewers often have helpful suggestions for data collection and analysis that strengthen our articles’ claims to have gotten it right.

What happens, however, if an article is published and a reader finds what they believe to be is an error? The APSR, like other journals, has grappled with several situations in which an article’s accuracy has been challenged after it has been published. While we do not have the broad evidence we would need to know for sure, we believe that one result of the movement toward more research transparency in political science has been an increase in the identification of errors in published works and related post-publication critiques.

In these Notes, we will discuss how the APSR handles errors and post-publication critiques and present our impressions of corrections and what they mean based on our experiences with them. In short, we believe that in an environment in which almost all scholars are accessing published work in electronic format, maintaining an accurate scholarly record is far easier than in the past. While we and other editors strive to ensure that the work published in journals is correct, we believe that the publication of corrigenda to correct errors should be understood as a reflection of our common aspiration as political scientists to provide the best answers we can to important problems with the tools that we have available to us. Of course, few people like hearing that they have made mistakes. However, if journals, critics, and authors can conduct conversations around criticisms with civility and respect, and if disciplinary understandings of corrections can shift to make them less stigmatized and more a part of normal political science, we believe that political science will benefit.

WHAT ARE POST-PUBLICATION CRITIQUES?

As we have explained in a previous Notes from the Editors (2023), we rely on guidance from the Committee on Publication Ethics (COPE) to develop policies addressing common dilemmas arising in the publication process. COPE urges journals to “allow for post-publication discussion on their site,” either permitting critics to submit corrective letters to the journal for publication or through another means, like a moderated forum. COPE advises journal editors about handling both formal and informal challenges to published works, but defines only the formal submission of a reader’s critique as a new manuscript to the journal as a post-publication critique. At the APSR, we have received occasional formal submissions of post-publication critiques, usually as a letter manuscript that raises concerns about a previously published work. COPE provides significant guidance for handling such critiques, which we discuss below.

COPE also acknowledges informal notifications about potential issues in a published manuscript that reach the journal through a means other than the submission of a letter or article for consideration. During our term as editors, we have received notifications about concerns in a variety of ways: individuals have contacted the journal to express their concerns, we have learned of criticisms through social media, or authors themselves have contacted us to alert us to problems they have discovered. We have a FAQ for readers who think they have found an error in a published article and are wondering whether they should submit a letter or contact the journal.

HOW DOES THE APSR ADDRESS POST-PUBLICATION CRITIQUES?

Regardless of the form that a post-publication critique takes, the journal carefully considers and evaluates any claim that the results in an article or letter are unreliable due to an error or ethical consideration. While the process unfolds a bit differently depending on whether a post-publication critique is a formal manuscript submission to the journal or a notification we receive directly or indirectly, the end aim is to ensure that the scholarly record is correct.

Of the 30 post-publication critiques or notifications (including notifications by authors themselves) that this team has handled, 15 have come in the form of a new manuscript submission to the journal that critiques a published paper. If the submission merely criticizes a published article or letter and does not allege that the results are unreliable, the journal handles it through the regular review process. If, however, the submission claims that a published article or letter has problems with reliability of the results or ethics, the editors undertake a more robust first consideration of the critique, determining whether it seems credible. If the critique does not hold up under closer scrutiny and the submission does not generally advance scholarly debate beyond the critique, the journal rejects the submission, and the matter ends there.

If the submitted critique is credible, we then consider whether the submission is only pointing out an error or alternatively presents claims that in our view advance scholarly debate or enhance understandings of significant substantive, theoretical, or methodological questions. If the potential contribution seems significant, the submission then proceeds through peer review process. A critiquing submission that survives this process will be shared with the critiqued authors when it reaches the point of conditional acceptance. We may invite the critiqued authors to submit a letter-length response (which will also go through peer review), a correction to their work, or both.

The journal has handled 15 notifications about concerns about reliability of results or ethics in a published manuscript that reached us through a means other than submission of a new manuscript. Of these, eight were initiated by authors, who contacted the journal because they became aware of an error in their APSR article or letter. In the other seven notification cases, the first step is to determine in a preliminary sense whether the concerns are meritorious. If the editors are persuaded that a problem may be present (or if we conclude that a formally submitted critique cannot be published but the concerns are nonetheless credible), we share the concerns with the original authors. We ask them to respond to the critique, offering them an opportunity to explain why their results are reliable or ethical concerns unfounded. We then review the concerns and the authors’ response and make a determination, sometimes inviting neutral third-party experts to assist in the evaluation. If the concerns are valid but minor, we invite the authors to prepare a correction, which is then published. If the concerns are more serious, the journal may consider issuing an Expression of Concern or even retracting an article or letter.

In any case in which individuals have contacted the journal directly to criticize published research, we inform these individuals of the outcome, whether or not any further action is taken.

NORMALIZING THE CORRECTION OF ERRORS AND POST-PUBLICATION CRITIQUES

We have sought to provide a transparent, reasonable, and fair system through which we can evaluate and assist authors to address problems in work that they have published in the APSR. Ideally, published articles would never have errors, but we do not live in an ideal world. The next best thing is to facilitate an environment in which errors can be corrected so that later readers of an article can cite and use it confidently.

Scholars are understandably passionate about their work and the energy they have put into it. Learning that something has gone wrong, despite the authors’ hard work, the editors’ careful consideration, and the scrutiny of peer reviewers, often through multiple rounds of review, is discouraging and may engender frustration. But in our experience, most corrections are minor. Going through the process of correcting an article and participating in a civil disagreement through a response letter can be instructive not only for the critiquer and the author, but for others who can learn from the exchange. We encourage a common commitment to achieve the most accurate and ethical political science that we can produce. We encourage individuals who identify errors to do so with some generosity of spirit, those correcting them to do so with openness and little defensiveness, and those reading corrections to view them as strengthening our common mission and not as stigmatizing for scholars who have responsibly corrected errors. In this way, we can collectively create an environment that facilitates moving forward when things go wrong.

The APSR is often at the forefront of emerging disciplinary norms, including around data sharing and research transparency. Research published in the journal is also often highly visible. This combination of visibility and transparency leads to greater scrutiny of research in the APSR. As a result, the journal receives many post-publication critiques or notifications about potential errors or ethical issues every year, the overwhelming majority of which are motivated by scholarly engagement and interest in ensuring published research is reliable and ethical. ¹ As editors, we have a responsibility to ensure that the scholarly record is reliable. This responsibility begins with peer review prior to publication but has become an increasingly normal part of the scholarly process after publication as well.