Go back

Reviewers of Covid papers play the good cop

    

The pandemic has turned reviewers from gatekeepers into coaches, says Serge Horbach

How should scientists communicate their research in a global crisis? How should they balance the need to generate and share knowledge with the importance of making sure that knowledge is robust? 

These have been major concerns since the spread of the coronavirus turned into a global pandemic in March 2020. Ever since, these questions have underpinned the scientific response to Covid-19. 

They lie behind top-down initiatives, such as journals’ efforts to accelerate publication—using fewer rounds of review, with shorter deadlines, and speeding up the production process—and funder mandates for depositing work in preprint servers prior to formal review and publication. Preprint publication was already the norm in some areas of research, particularly physics and mathematics, but it has quickly spread to encompass large parts of science. 

These changes are well documented. More subtle are the bottom-up, emergent changes caused by the pandemic in how scientific results are communicated and validated. These were not imposed on researchers but seem to have been driven by a collective shift in attitudes.

Inspired by the rapid acceleration of review processes at biomedical journals, I set out to study how else the pandemic might have changed peer review. To do this, I took advantage of the open peer review model of two journals, the British Medical Journal and eLife, to compare the content of review reports and editorial decision letters for Covid-related and non-Covid-related manuscripts

I found that even though Covid-related papers were reviewed much more quickly than those on other topics, there was no loss of thoroughness. I did see, however, some remarkable shifts in the quality criteria used to assess these papers. 

Changed priorities

Reviewers of Covid papers were less likely to ask for additional data or experiments, or for major restructuring. Instead, where they identified deficiencies and gaps they tended to ask for conclusions to be modified and qualified. Reviewers of Covid-related papers also seemed to place greater emphasis on the social and medical relevance of research, rather than on its pure methodological soundness or novelty.

All in all, this suggests the crisis has seen an emphasis on getting work out there as quickly as possible. This touches on a long-standing debate on the function of journal peer review and the task of reviewers. 

Reviewers act as gatekeepers, excluding research below a minimum standard. But they can also be coaches, helping to make a study as good as it can be. For work related to the pandemic, the balance seems to have swung towards the latter role. 

This shift is understandable in the context of the pandemic. It shows that researchers’ instinctive response to the crisis has been cooperative, fuelled by a desire to meet the challenge collectively. But it also raises questions about the quality and reproducibility of Covid research. 

The first few months of the pandemic saw several high-profile retractions, such as a paper in the Lancet on treating Covid-19 with the malaria drug hydroxycholorquine. These sparked debates about the functioning of science and its self-correcting mechanisms. They threatened to damage public trust in science at a moment when it was vital. 

Such cases showed once again that poor science is still poor science, even if reviewers insist on it being hedged with caveats. Acknowledging a paper’s limitations rather than fixing them might not be enough to ensure its value or trustworthiness. 

The pandemic’s long-term, large-scale effects on how research is communicated and evaluated remain to be seen, but they are likely to be significant. There are hints of a shift from a system based on blind review and journal publication to one based on preprints, followed by informal social media scrutiny, and then journal submission and review. 

Some of the most high-profile and influential Covid-19 papers have been reviewed in a remarkably open and transparent way, unmediated by publishers or editors. New, community-based platforms are emerging within the scientific ecosystem, and traditional journals are being forced to rethink their roles. 

Some of these trends, such as biologists’ growing use of preprints, were already underway but they have been turbo-charged. How this changes the functions and operation of both peer review and journals—and, ultimately, how this affects science—remains to be seen. Rather than sitting back and watching what happens, this seems like an ideal moment for the research community to actively steer developments in the direction it wants. 

Serge Horbach is a postdoctoral research in the Danish Centre for Studies in Research and Research Policy, Aarhus University

This article also appeared in Research Europe