The assessment’s architects face many questions on what to value and how, says Grace Gottlieb
Research culture covers a vast breadth of areas, from career pathways and incentives to research integrity and equality. Most of these aren’t easily measurable—and even if they were, there’s a limit to universities’ capacity to measure them all.
So while the increased emphasis on culture in the 2028 Research Excellence Framework has been broadly welcomed, the question of how it will be assessed has also created apprehension. How will the funding bodies decide what areas of the research system are worth measuring and what the best metrics are?
All involved in the REF recognise the work needed to develop indicators of research culture. The sector has an opportunity to inform these decisions; the funding bodies are interested in learning from institutions, particularly those developing their own key performance indicators to measure culture change.
Performance vs progress
Aside from deciding what aspects of culture to measure, there’s another question about values to answer: when measuring research culture, is performance or progress more important?
The answer depends on where you set the balance between seeing the REF as a mechanism to reward performance and seeing it as a way to incentivise behaviour change. Steven Hill, director of research at Research England, has indicated that culture change, and the direction of change, is probably more important than absolute performance, or at least needs to be considered alongside it.
This raises more questions, around how to quantify progress, on top of those around how to quantify performance. How this balance is struck could have a real impact on different institutions’ success on research culture metrics.
A focus on progress would arguably do more to accelerate culture change. But could focusing on change and its direction disadvantage institutions with histories of investing in research culture, or perhaps make it more difficult to distinguish between institutions? There is already uncertainty around how institutions can set themselves apart from the pack in the People, Culture and Environment element of the REF.
Given all this, a comprehensive process is needed to select the right indicators to measure culture. Ideally, this would be iterative, enabling us to try different options and figure out what works over time.
But is there time for this? While the funding bodies’ consultation this coming autumn will be hugely important, there will be a limit to the testing that can be done before REF 2028. The availability of data and practicality of different metrics will also be factors.
Inputs and outputs
The changes to the REF to assess research culture have also stirred controversy because they shift to rewarding inputs to the research system rather than focusing on outputs and outcomes. On the other hand, well-trained researchers are an important output of the research ecosystem, and one could also see culture as an outcome of research assessment practices.
There seems to be a distinction developing here between the outputs and outcomes of research and those of the research system. In the context of the latter, broader conception, it’s possible to see research culture as both an input and an outcome.
A comprehensive view would consider policies alongside recognition and reward systems and the practices they seek to promote, including collegiality, mobility to and from other sectors, research integrity and openness.
Culture touches even on the narrower definition of the outputs of research itself. Broadening the notion of research outputs has been a longstanding challenge. The proportion of outputs not related to publications or books fell from 6 per cent in the 2008 Research Assessment Exercise to 2.4 per cent in REF 2021.
This has prompted the Hidden REF campaign to drive academia to recognise and value the full diversity of research outputs. It recently launched a 5% Manifesto, calling on institutions to ensure that at least 5 per cent of the outputs they submit to REF 2028 are in non-traditional formats.
As the REF broadens its view of the research system, this raises yet more questions. How should the value of, say, a highly cited paper in basic research compare with a piece of software that underpins a range of subsequent research, or the training of a PhD student who goes on to work in industry?
The more the funding bodies seek to make assessment reflect the variety of research endeavours, the more decisions they must make on how to measure and weight their value. Given the REF’s huge influence, these decisions will shape the behaviour of institutions, researchers and the whole ecosystem. Time will tell whether REF 2028 lives up to the big ambitions behind it.
Grace Gottlieb is head of research policy at University College London. This article is based on the event The Emerging Shape of REF 2028, organised by the Foundation for Science and Technology, UCL and the Research on Research Institute, in association with Research England
This article also appeared in Research Fortnight