Proposals for REF 2028 are quietly revolutionary, say James Wilsdon, Stephen Curry and Elizabeth Gadd
Six months after it started rolling, the Coalition for Advancing Research Assessment is gathering momentum. Almost 600 organisations in more than 40 countries have signed up to its underpinning agreement, with dozens more joining each week.
In mid-May, the San Francisco Declaration on Research Assessment, on which Coara builds, marked its 10th anniversary with events around the world. At a national level, there are ongoing or proposed reforms to assessment frameworks in Australia, Czech Republic, Italy, New Zealand and Sweden.
Even China is making a renewed drive to break the power of the ‘four onlys’—only papers, only titles, only education background and only awards—over the funding, hiring and promotion of researchers.
Today, the four UK higher education funders add their voices to this swelling chorus, with the publication of initial decisions on the design of the 2028 Research Excellence Framework. Alongside these, an advisory group chaired by Peter Gluckman, president of the International Science Council, has published its independent audit of the UK’s assessment system.
Inside number nine
Although the REF 2028 document includes points for further consultation, this essentially marks the end of the two-year Future Research Assessment Programme (Frap). The direction is set, with finer details to follow.
Since 1986, UK universities have lived through eight cycles of national research assessment. These have evolved and become more complex: REF 2021 involved 157 institutions submitting more than 185,000 research outputs and 6,700 impact case studies from 76,000 staff.
This has given the REF a rather daunting reputation globally as a framework to study, but not necessarily emulate. So the Frap’s outcomes will spark interest here and overseas.
If not the total revamp that some hoped for, today’s conclusions are quietly revolutionary. They continue the process that began with REF 2014 of broadening evaluation from being primarily about research outputs (rebranded in the new report as contributions to knowledge and understanding, to be more inclusive of datasets and other formats), to placing more weight on societal impacts (now engagement and impact) and an expanded category of people, culture and environment.
In 2028, outputs will account for 50 per cent of the assessment, down from 60 per cent in 2014. This falls short of the Gluckman review’s call for the three pillars to be equally weighted, but nevertheless represents a shift towards embracing research culture and practice as vital to any concept of research excellence. It also makes equal weightings a stronger possibility for the cycle after REF 2028.
The human touch
From our perspective, as co-authors of the 2022 report Harnessing the Metric Tide, commissioned to inform the Frap process, there is a huge amount to welcome. In particular, three aspects seem particularly positive.
First, a refreshed trio of purposes clarifies what the REF is trying to achieve. While care will be needed to prevent the stated aim to provide “insights into the health of UK research” turning into a licence to rank, a clearer sense of purpose will shape discussions of costs, benefits, opportunities and burdens. A firm date for the next assessment also gives universities a definitive timetable.
Second, in line with our advice on the responsible uses of metrics and a separate analysis of the potential for using artificial intelligence in research assessment, led by Mike Thelwall, the REF’s architects have wisely decided not to opt for a fully automated or metrics-based approach.
These and other methodological options will be kept under review, which also makes sense, particularly if accompanied by investment in the open infrastructures needed to support intelligent and responsible uses of quantitative data.
We also welcome the commitment to identifying and consulting on new “data for good” to help recognise progress under the people, culture and environment pillar of REF 2028. This element will be further strengthened by the adoption of a structured narrative format for submissions, following a planned consultation on the details.
No more individuals
Third, changes to the method for measuring the size of submissions are big news. These will sever the link between individuals and submitted outputs, finally fulfilling one aim of the 2016 Stern Review.
By using an average measure drawn from the data that universities routinely return to the Higher Education Statistics Authority, this rolls the process of identifying staff with significant responsibility for research, the criteria for REF submission, into business as usual, rather than making it a one-off exercise.
So farewell minimum and maximum numbers of contributions per individual, and hello to fully aggregated contribution pools for each unit of assessment. Implemented with care to ensure that the spread of an institution’s research is still fairly represented, in line with the report’s wider principle that “supporting a healthy research culture should be an underpinning principle of the REF”, this may strengthen the emphasis on positive research environments and the team-based nature of much high-quality research.
So, overall, this is a positive package. The four UK funding bodies—and particularly the team at Research England led by Jessica Corner, Steven Hill and Catriona Firth—deserve congratulations for steering a complex process of evidence gathering and consultation towards a set of radical yet pragmatic conclusions.
They could have moved further and faster towards formative modes of assessment aimed at shaping the system as well as surveying it, in line with more ambitious signals in the Gluckman review. And they might, as our report suggested, have offered a longer-term roadmap for change over multiple REF cycles. But you can’t win ’em all, and even the REF’s fiercest detractors should find reasons to give a clap or two to the Frap.
James Wilsdon is director of the Research on Research Institute and professor of research policy at University College London. Stephen Curry is professor of structural biology and assistant provost for equality, diversity and inclusion at Imperial College London, and chair of the Declaration on Research Assessment steering committee. Elizabeth Gadd is research culture and quality lead at Loughborough University, and vice-chair of the Coalition on Advancing Research Assessment.
On Wednesday 5 July, UCL, in partnership with the Foundation for Science and Technology, the Research on Research Institute and Research England, will host an open forum on The Emerging Shape of REF 2028. The event is free to attend in person and online.
A version of this article appeared in Research Fortnight