Research agencies take note: a respected think tank confirms peer review is no guarantee of excellence
Peer review, so deeply embedded in research today, has become synonymous with the idea of excellent or even just ‘reliable’ research. Yet at the same time, we know that a great deal of valuable research has never been peer reviewed. Moreover, a good number of peer-reviewed projects contribute little or nothing to the sum of human knowledge.
Research administrators need to realise that peer review, though unquestionably useful, does not of itself ensure quality and originality in research. Peer review has emerged as the dominant approach for European research agencies selecting projects—but that doesn’t mean they should give up on finding alternatives, in appropriate circumstances.
The message is brought home in a report, Alternatives to Peer Review in Research Project Funding, issued this week by the think tank RAND Europe. It argues that many of the assumptions we routinely make about the strengths of project peer review are, in fact, unproven.
RAND Europe is an offshoot of the California-based RAND Corporation, long-time purveyor of new concepts to the Pentagon. The message of its report probably reflects the practices of perhaps the best-known counterexample to peer review: the United States government’s Defense Advanced Research Projects Agency. Darpa has usually chosen projects by hiring smart project managers and then giving them more-or-less complete discretion (shock! horror!) on what to fund.
The Darpa approach is widely considered a success (although critics charge that the agency isn’t what it used to be). The context is significant, of course: to the military, for example, effective solutions to the problem at hand matter more than the niceties of a transparent, accountable project-selection process. Darpa’s track record has nonetheless drawn imitation, at the US Department of Energy—which has set up its own Arpa—and, to a lesser extent, at the US National Institutes of Health.
These developments in the US reflect a growing debate on whether peer review is really all it’s cracked up to be. As the RAND Europe report points out, its advantages are hard to prove, whereas its disadvantages are clear. The process of peer review, almost by definition, involves established elders selecting future research paths. These paths will tend to be conservative. At some agencies, it is often said that you’re safer to know the findings before you even put in the application.
In many parts of Europe—and in the research programmes of the European Commission—the priority for now is to build peer review up, not to knock it down. When research funds have been distributed, in the past, using methods other than peer review, the funds have often been wasted.
It’s only after the principle of peer review has been thoroughly embedded in the system (partly as a guarantee against corruption and nepotism) that the time comes to look beyond it, for approaches that will extend the boundaries of inquiry. That time has now arrived in the US and in parts of northern Europe. Alternatives to peer review will be sought by research agencies and philanthropic organisations that are already on the cutting edge, and want to stay there.