The integrity of the research record: a mess so big and so deep and so tall

Abstract

Summary

Poor research integrity is increasingly recognised as a serious problem in science. We outline some evidence for this claim and introduce the Royal College of Psychiatrists (RCPsych) journals’ Research Integrity Group, which has been created to address this problem.

The extent of the problem

Many research findings are incorrect, even if the studies are carried out completely correctly and all are published.¹

Sadly, only about half of all registered clinical trials are ever published² and the published half may well overstate benefits and understate harms. These figures are unknown for observational studies, but are likely to be worse because of the lower costs and the lack of requirement for registration.

‘Questionable research practices’ (QRPs) are common, including selective reporting, outcome switching, ‘p-hacking’, the seemingly low-consequence ‘gift authorship’ and the fabrication of data.³ These exist because of the tendency for journals to accept papers with eye-catching results over those with more moderate claims.⁴ Thus, published research reflects more the criteria required to achieve publication than scientific accuracy. These problems extend to systematic reviews, which carry the double burden of summarising an already compromised literature and being subject to these forces themselves.

Research fraud has been detected in about 5% of papers, just from examining figures in the papers themselves.⁵ In our view, is likely that only a minority of research fraud is detectable this way, so the total figure may be higher. Recent examples, since retracted, include a paper of a trial of hydroxychloroquine for COVID-19 published in The Lancet and a paper about risk factors for mortality in COVID-19 published in the NEJM. Less recently, but still importantly because of its significant ongoing public health harms and its mental health subject matter, is a paper falsely linking certain vaccinations given to children in the UK to the causation of autism. This was retracted from The Lancet after 11 years of deliberation.

The pathway to the current problem

How did we get here? There is strong competition for advancement in academia: only 0.5% of people in the UK with PhDs ever become professors and in the USA only 12.8% achieve academic tenure. Academics and institutions are harshly judged on their outputs against others using dubious measures of research quality such as the impact factors of the journals in which they publish their work. Careers and livelihoods depend on doing well on such metrics. Striking, positive findings increase the probability of a paper being accepted by such journals.⁴ Alongside this, journals are similarly in competition to publish work which will be cited well. Thus, papers in prestigious periodicals have a higher risk of retraction than papers published elsewhere, when one might expect a prestigious journal to have greater pre-publication scrutiny.

Yet institutional leadership at universities and learned societies persists in practices that generate and maintain these perverse incentives, which slow scientific progress, offer poor value for money for taxpayers and other funders, and ultimately harm our patients. Recently, mercifully, aggressive research environments have been identified as needing attention and reform.

Thus, there are strong incentives for researchers to do wrong rather than right. Some researchers have left the sector, disillusioned and morally injured, after realising that to succeed in their careers, they need to undermine the very values that led them to become researchers in the first place.

Altogether, these issues have very serious consequences for the state of knowledge and the credibility of all scientific output. This has an impact on the general population, being the ultimate consumers of the results of research and healthcare informed by those results.

The RCPsych's response

So, what can a single learned society and a small group of scientific journals do to address this enormous problem?

We must accept the pervasiveness and importance of poor research integrity. Accordingly, we have created a Research Integrity Group of the Editorial Boards of the BJPsych, BJPsych Open and BJPsych Bulletin, primarily to oversee investigations following allegations of poor research integrity in the journals’ published output. We are making this available as a resource to BJPsych Advances and BJPsych International as well. The work of our group will be of value generally to the College's activities, especially where there are disputes of evidence.

At a system level, we have tightened the initial checks on submitted articles and research papers. All the details of trials are now carefully checked, including that the ethics permission and pre-registration were complete before recruitment began. Automated plagiarism screening is now undertaken on every submission as well. Evidence of outcome switching or poor adherence to protocols results in rejection, whether these become apparent during the initial checks or later. We also now support the use of preprint servers. Within this, we are careful to preserve the pluralism of approaches necessary in our specialty and to draw distinctions between scientific weaknesses, which are inevitable, and poor practice, which is not.

A reasonable concern is that any paper rejected by the RCPsych journals may still be published elsewhere. However, we cannot control that, and high standards are necessary to ensure quality publications in our journals and to lead the development of scientific publication.

As well as serving to oversee investigations following expressions of concern over research integrity matters and working to improve journal and College processes, the Research Integrity Group is linking with other efforts to improve research integrity: we have met with prominent campaigners and managers from other journals to look beyond the immediate problems to reinforce efforts to tackle the perverse incentives that create, nourish, tolerate and protect poor research practices across our sector.

We are trusted to care for people often at the lowest points in their lives. We must bring the best evidence to bear on their individual difficulties. By making those areas over which we have influence as good as they can be, we can honour the motto of our College and ‘Let Wisdom Guide’.