Ratings from journal editors make peer review more effective and inclusive, says Laura Feetham-Walker
For publishers, it’s easy to recognise a good review when we see one: they’re clear, detailed and useful, making the choice of the next step in the process clear.
However, journal editors have tended not to give reviewers any feedback on the quality or usefulness of their reports, let alone pointers on how to improve them. For early career researchers in particular, this lack of communication makes it harder to develop skills and confidence as a reviewer.
At IOP Publishing (IOPP), which is wholly owned by the UK’s Institute of Physics learned society, we surveyed 1,200 researchers who had reviewed or been invited to review with IOPP between January 2018 and March 2020. The results showed that respondents saw feedback as a way to improve the experience of reviewing and the most valuable form of recognition for their efforts.
Feedback might be notification about the final decision on the paper, comment on the quality of their report, insight into other reviewers’ comments or details of post-publication metrics.
The survey prompted us to trial a reviewer feedback system on three IOPP journals from 2022. It received an overwhelmingly positive reaction, with more than 85 per cent of reviewers involved in the trial saying they found receiving feedback on their report useful.
Feedback rollout
Last month, after taking time to integrate the new system with our pre-existing workflows, we rolled out the option for reviewers to receive feedback on their reviews across all 58 IOPP-owned journals. Reviewers who opt in will now receive an evaluation of how useful their work was on a scale of one to five, with five being outstanding and one indicating that the review was not suitable to inform an editorial decision.
Reviewers can get more detailed information on the score, which matches ratings to factors such as level of detail, thoroughness of comparisons with the existing literature, or unethical or inappropriate comments. Any reviewer who submits a five-star report, regardless of whether they opt to receive feedback, receives IOP Trusted Reviewer Status.
The feedback is produced by our in-house editors, who consistently evaluate reviewer reports based on their insight and constructiveness during the editorial decision. Initially, this was a way to collect useful information about the peer-review system and identify excellent reviewers. Now this process is much more open.
Positive response
Reviewers can also provide their own feedback. There were initial concerns that some researchers would not welcome evaluations of their reviews, but the response so far has been overwhelmingly positive, with many saying that receiving comments on their efforts would help them to improve as reviewers.
Over the past three years, IOPP has worked to diversify its reviewer pool and involve people from places and groups who have traditionally been excluded from the process. We believe that the secret to the success of these innovations has been that they are not prescriptive.
Instead, all the different options and changes to peer-review programmes—including the reviewer feedback and certification systems—are configurable by reviewers. Everyone can opt in or out to whichever features they find most and least useful.
Rating reports is just one step that publishers can take to support reviewers, helping them to improve both their skills and the manuscripts they assess.
IOPP has also introduced process to recognise co-reviewing and offer peer review training and certification as ways to support the next generation of reviewers.
Strengthening the process
Peer review is the backbone of robust and trustworthy research. Peer reviewers are specialists in their fields, bringing a perspective that allows them to scrutinise, assess and validate the quality and originality of research, helping to ensure the accuracy of the scholarly record. Creating opportunities and systems to provide reviewers with feedback is one way for publishers to strengthen the peer-review process, make review more inclusive, and contribute to the advancement of scientific knowledge and innovation.
Giving reviewers such feedback bolsters a system that has faced concerns about both the volume of work involved and the validity of reviewing. I would like to see scholarly publishers across disciplines consistently offer feedback to their reviewers, as a means to increase the quality and transparency of peer review across the sector.
Laura Feetham-Walker is reviewer engagement manager at IOP Publishing, based in Bristol, UK
This article also appeared in Research Europe