It would appear that scientists are human and subject to the frailties of the rest of the community. A recent article and editorial in the Economist “Trouble at the lab” (19 October 2013) has highlighted some of the ways in which scientists have been failing to uphold the ideals that give science the public respect and belief that is lacking for many other professions. Using research of economic importance - so especially medical, biotech and pharmaceutical research - they have generalized their criticisms to all science. In their view scientists are “doing too much trusting and too little verifying, to the detriment of the whole of science and of humanity”. Citing various notorious papers as well as some recent surveys and attempts to reproduce the results in landmark papers, they concluded that we are, as a group, unlikely to verify the experimental work of others as this does not constitute novelty (a key criteria for publication in many journals), unlikely to publish negative results, and are focussed on discovery rather than verification. In addition we apparently plagiarise, suppress data, falsify data, use inappropriate statistical tools, poorly design our experiments and assume that peer review will detect and correct all these failings before publication.
In the millions of papers published there are bound to be some with false data, some with plagiarised ideas and many with less than ideal experimental design. What is disturbing is the contention by the Economist that surveys in some disciplines show a lack of adequate understanding of statistics by both authors and reviewers, the unavailability of original data to determine its quality, and flaws in computer modelling that are hard to unearth without examining the original code. In short peer reviewers must take a lot on trust, and again there is evidence that despite multiple reviewers, the peer review process itself is far from perfect. Couple this with a determination to publish little and often to meet tenure and promotion expectations and we have reason to be concerned. Their remedy is to require funders to pay for reproducibility studies, require journal editors to publish negative results, and ensure researchers are taught basic statistical and design skills and judged on the quality, not the quantity, of their work. In addition the Economist suggests that reviewers should insist on having access to the original data and repeat the original statistical tests to check the accuracy of the analyses. Indeed the article goes on to suggest that researchers should have open on-line notebooks, very detailed methodologies freely available and an enthusiasm for publicly acknowledging their mistakes.
There are certainly problems in particular fields where money and politics loom large - such as medicine and pharmaceuticals. But for the community served by this journal many of the problems have little immediate relevance. Of course we support the idea of free access to data, the acknowledgement of errors and the need for reproducibility checks on a sample of methods. However, reviewers are already heavily burdened and cannot be expected to repeat the statistical treatments themselves when assessing a paper and funding is tight making verification much less attractive than novelty.
Science has its own ethics and values and journals can and do play an important role as gatekeepers in the system. There is no harm in reminding both authors and reviewers that it is not the protection of reputations that is important but the quality, accuracy and reproducibility of the endeavour that is at stake. Verifying results and acknowledging mistakes are important parts of this. There is no room for complacency but equally there is no need for wholesale change. We are not perfect but we are trying.
