Paying researchers to referee papers is unlikely to result in better peer review, but enhancing the prestige and visibility of reviewing might, argues John Whitfield.
Dip into the academic literature, press or blogosphere, and it’s not hard to come across the feeling that peer review is not working as it should. Both sides of the transaction seem to feel hard done by. As authors, researchers think that reviews are often slow to come, and can be ill informed and vindictive when they do. As referees, they fear that assessing the work of others will take up time that could be better spent on their own work, and some resent doing unpaid labour for commercial publishers.
An option that has been kicking around for at least a decade is to pay reviewers, rewarding them for their time and providing an incentive for swift, high-quality reviewing. Journals published by the American Economic Association already pay $100 (£64) for “timely reports”. A recent start-up, Rubriq, which aims to speed up reviewing by charging authors for pre-submission reviews and brokering links with journals, pays referees $100 for responding within a week.
One issue in understanding the problems of peer review and the merits of proposed fixes is a shortage of hard evidence. In a study published in Research Policy in February, sociologist Flaminio Squazzoni of the University of Brescia, Italy, and his colleagues contributed to the debate by testing the notion that hard cash makes for better reviews.
The researchers devised an experiment that mimicked peer review. Player A was given some money, and had the choice of passing all, some or none of it to Player B. The researchers trebled what was passed on, so that B received three times A’s gift. B then had the option of returning all, some or none of their windfall to A.
So if A hands over €10, B gets €30. If B then splits that down the middle, each person ends up with €15.
Both players, then, can profit if they trust one another and share fairly. But B also has the option of exploiting A; knowing this, A may be dissuaded from sharing at all, so everyone loses. The researchers see A’s position as like an editor putting resources into attracting the best papers; the beneficiaries of this investment, the authors, have the option of either honouring that commitment or submitting poor work.
Any two participants encountered each other only once, but each played many rounds with different partners. To mimic the reviewers’ role, in some rounds a third person informed the investing ‘editor’ whether the returning ‘author’ had been sharing in the past. Editors could then use this information to decide how much to invest in the author, giving authors a reason to develop a good reputation.
Not surprisingly, adding an unpaid reviewer prompted authors to give more back, which encouraged editors to invest more. But when reviewers were paid a fixed amount, this effect became far smaller, with editors giving authors no more than they did in the absence of a reviewer. Giving reviewers a financial interest in the process seemed to make editors trust them less.
The researchers attribute this effect to something called motivation crowding: the idea that if you give people many incentives, some will drown out others. In particular, there’s good evidence that replacing a powerful sense of social duty with a small amount of money is a bad idea. Paying women to donate blood, for example, reduces donation rates by almost a half, and Israeli economists found that fining parents for not picking up their children from nursery on time resulted in increased tardiness, with the moral imperative gone and the fine seen as a payment for some extra childcare.
The latest study is not a perfect reproduction of the social dynamics of peer review—usually, for example, authors make contact with editors, not vice versa. But that need not be fatal: experiments like this are useful because they reveal people’s traits and biases, not because they simulate the world outside the lab.
As someone who has played the author-editor game numerous times from both ends, I can see the merit in using payment to enforce deadlines. But—brace yourselves—paying for work does not guarantee against it being shoddy. Changing reviewing from a social service to low-paid drudgery could well harm its quality.
Besides, paying people to review things is so 20th century. The web has turned us all into reviewers, to the extent that professional critics have developed a sideline in writing obituaries for their trade. The torrent of freely offered opinion, tapped by sites such as Amazon, hints at what might be a better way to improve peer review: increasing the social rewards, and building on the sense of reciprocity and duty that motivates referees. Rather than money, Amazon and its ilk reward their contributors with visibility and prestige through devices such as top-reviewer charts. They have found ways to incentivise, capture and distribute intelligent word of mouth.
Along these lines, Squazzoni and his colleagues argue that publishers and funders should be honouring referees by, for example, naming them on published papers, appointing the best to their editorial boards, and considering excellence in reviewing when assessing grant applications. Like teaching and mentoring, reviewing is an essential contribution to academic life that assessments focused on published papers do not pick up.
The biggest block to recognising reviewers’ work is, of course, that reviewing has traditionally been anonymous, allowing researchers to rubbish their colleagues without harming relationships or career prospects.
But perhaps anonymity is a norm that no longer serves the review process well. When fields were small, and a few grandees could handle a journal’s entire load in a particular subject, strong social ties gave researchers good reason to behave conscientiously as referees. Even with anonymous reviewing, disciplines were village-like, with everybody knowing everybody else’s business. In such an environment, people feel social pressure even when no one is watching them.
Now, each academic discipline is more like a city. A reviewer is less likely to be friends with the people whose work he or she is judging, and more likely to be competing with them to publish similar findings in similar journals. This combination of weaker social ties and stronger competition lessens referees’ motivation to do the right thing.
Anonymity might not be healthy for editors’ integrity, either: in March, the Committee on Publication Ethics called for an end to the practice of journal editors reviewing submissions themselves rather than bothering to find an external reviewer.
Controlled trials of open review have found that it encourages referees to be civil and make criticisms constructive, although it also makes comments more positive and seems to make reviewing slower. Following randomised controlled trials, the British Medical Journal has mandated open reviews for more than a decade with, according to its evidence to the House of Commons Science and Technology Committee in 2010, “no significant problems”. The journal also rewards the best reviews with credits towards medics’ continuing education.
Other journals that have experimented with signed reviews, such as PLOS Medicine and Nature, have found that most referees prefer to remain anonymous. A welter of different approaches to openness has been and is being tried, particularly by open-access, online-only journals: reviews being revealed to authors or published alongside papers, referees volunteering or being picked by editors, and review occurring before or after publication.
There is no sign of an accepted alternative to anonymity emerging, and it may be that different models suit different publications. Referees seem less willing to reveal themselves in higher-impact journals and more competitive fields; in its evidence to parliament, the BMJ suggested that its move into open reviewing has succeeded where others failed because clinical medicine is less cut-throat than biomedical research.
What seems to be lacking is an attempt to combine social rewards for openness with rewards for good reviewing. Publishing reviews alongside papers builds recognition for the craft of reviewing, and adds huge value for readers, but does not solve the question of how to reward insightful negative reviews that lead to rejection.
Ultimately, journals can only do so much to change researchers’ attitudes towards peer review and how to behave as reviewers. If researchers think the current system is broken, they are the ones who need to fix it, as they are the ones with most to gain or lose. If referees are willing to expose themselves for the greater good, journals should try to find ways of giving them kudos for doing so.
Until and unless open review becomes the norm, there are other options that preserve anonymity while recognising reviewers. Alongside payment, Rubriq gives reviewers a record of their work, to “show [their] additional contributions to the literature in [their] field”. In March, 35 Elsevier journals began piloting a scheme to award certificates of excellence to the editors’ favourite referees. And in January, Molecular Ecology named its 300 best reviewers, the top 8 per cent, measured by speed and workload over the preceding two years.
Tim Vines, Molecular Ecology’s managing editor, has suggested that unique identifiers devised to assign authorship, such as Open Researcher and Contributor ID, could be adapted to measure reviewing activity across journals. This is perhaps the most promising solution of all: as well as measuring a reviewer’s productivity, it would provide a proxy for quality, as the best referees are likely to be those asked to do the most reviews.
Care would be needed in the design of such a metric. For example, a reviewer’s score ought to decay with time, to encourage long-term commitment.
But, done well, such a measure would have multiple benefits. Academics would be able to see who is doing their bit and who is freeloading on the efforts of others. Learned societies would have a way to honour the top reviewers in their fields. And funders and research assessments, which have power over researchers’ careers and an interest in a well-functioning publishing system, would have a way to assess and reward reviewing. With such incentives, researchers might soon be cultivating their scores as zealously as eBay traders court positive feedback.
John Whitfield is the comment and analysis editor of Research Fortnight and the author of People Will Talk, a book on the science of reputation.
Something to add? Email comment@ResearchResearch.com