The Royal Society believes that assessment of research quality must focus on institutions if it is to create the right incentives, says Ottoline Leyser.
A central purpose of the Research Excellence Framework is to determine the size of the funding councils’ block grant to each university. The grant is awarded to institutions, so it makes sense that the REF should focus on assessing institutions in the round.
That is why the Royal Society is today calling for the REF to shift its emphasis from the work of individual researchers to a portfolio approach, illustrating the breadth and depth of an institution’s research environment.
The REF already evaluates research environment—it accounted for 15 per cent of the mark in the 2014 evaluation. Universities submitted a narrative statement explaining how they created a high-quality research environment and, in parallel, research outputs and examples of wider impact.
These elements of the current system are entirely appropriate. However, the interaction between the REF’s rules and other pressures on universities creates substantial problems.
In particular, the system centres on the inclusion or exclusion of individual academics. Most of those included must submit four research outputs produced during the assessment period, which are scored.
Their summed scores are the biggest single contributor to the REF outcome. A university’s block grant is a product of its REF score and the number of academics submitted. In effect, the system monetises academics with four top-rank outputs.
This has led universities to take a risk-averse approach to selecting outputs, focusing on papers published in high-ranking journals. The definition of excellence is limited to world-leading discoveries, rather than taking in the breadth and depth of activity needed to drive a sustainable and effective research endeavour.
It is not surprising that universities have appointed more and more academics with the ‘right’ sort of outputs. Staff not fitting the mould are undervalued and crucial activities have been neglected. For example, papers describing negative or confirmatory results are unlikely to score highly in the REF, so there is little incentive to write them up—meaning that others will repeat dead-end experiments or unnecessary replications.
Last year’s Stern review identified these perverse incentives. Its proposals, including making the REF more institutionally focused by breaking the link between individual academics and their outputs, form the basis of a consultation by the Higher Education Funding Council for England that ends today.
However, the proposed solutions are still focused on submitting individual researchers. This immediately revives questions about who they should be and how many outputs each should submit. The link between academics and their outputs remains.
No researchers should be submitted to the REF. None. All that is needed is a measure of research volume, to determine how many outputs should be submitted in each subject area and to calculate the size of the block grant.
Although not a trivial task, such a measure could be derived from data already collected by the Higher Education Statistics Agency. It could include academic staff, support staff and researchers at all career stages. There would be no need to decide who qualifies as research active or independent, who is employed on a specific day, or who has special circumstances.
For each subject area, this number would dictate how many research outputs should be submitted. Only outputs where a significant part of the research was conducted at that university should be eligible. Under these rules, multiple universities could submit the same output.
These changes will remove many of the problems associated with submitting individuals, but they won’t address the corrosive focus on particular types of output.
A high-quality research portfolio doesn’t consist only of groundbreaking discoveries. It contains work that cements initial findings, integrates existing work, develops tools and resources such as databases, engages diverse groups and nurtures the next generation of researchers.
Some of these elements feature in the statements on research environment, but they are absent from the output portfolio. They should not be.
REF panels should go beyond simply judging outputs, one by one, in isolation. They should provide a holistic assessment of the portfolio of outputs, recognising the importance of breadth and depth in output types. Such an approach would more robustly assess the quality of the research environment built by each institution, ease pressures on careers and morale, and provide the incentives for diverse, interdisciplinary and collaborative work.
Discussions with researchers show how deeply embedded our assumptions about the REF have become. Debate has focused on how fairly to include everyone, when it should have been about how fairly to include no one. Only a system with this as a fundamental principle can deliver a REF that is fit for purpose.
Ottoline Leyser is chairwoman of the Royal Society Science Policy Advisory Group and professor of plant development at the University of Cambridge.