Go back

REF 2021: ‘case-study cliff edge’ suggests gaming

Panel member fears that staff contracts may have changed to fit in with REF requirements

Patterns in case study submissions to the Research Excellence Framework 2021 suggest that institutional “gaming” still occurred despite changes to the rules, a research consultant has claimed.

Writing in Research Fortnight, Simon Kerridge reports that the number of staff reported in submissions show peaks just before the size that would require additional impact information from institutions.

Kerridge, an independent research consultant and adviser to the REF 2021 Main Panel C, which assessed research in social sciences, believes this “case-study cliff edge” is suspicious.

“It seems that, despite the rule to submit all eligible staff, many submissions adjusted their numbers to be just under the threshold needed for an additional impact case study,” he writes. “This can be seen as institutions gaming the REF.”

He adds, “My fear is that some people will have been persuaded, one way or another, to have research removed from their contracts or cut their hours, perhaps even to zero, in the name of optimising a REF submission.”

Changes to the REF rules were made after the 2014 exercise to avoid institutions “gaming” the system to improve their quality ratings. Unlike the REF 2014, in which universities could choose who they submitted, for the REF 2021 universities were required to submit all eligible staff for assessment.

But Kerridge said he fears that gaming had still taken place. Submissions to REF 2021 with less than 20 full-time equivalent (FTE) staff needed two impact case studies, while an extra impact case study was required for each additional 15 FTE up to 110.

In an analysis of the REF submission data, Kerridge pointed out that “submission sizes peak just below case-study thresholds”, with a submission seven times more likely to be within one FTE below the amount of staff that would trigger the need for an extra impact case study, than within one FTE above it.

For example, there were 81 submissions between 19 and 19.99 FTE, compared with 15 in the 20-20.99 FTE range.

A spokesperson for UK Research and Innovation, the parent body for Research England, which is one of the four UK funding bodies that runs the REF, said: “There was flexibility for universities to map all staff with significant responsibility for research into submitting units for return, according to the best presentation of the research being carried out in that institution. Because there is some overlap at the boundaries between Units of Assessment, there may a choice as to where to submit the work of some staff with significant responsibility for research.”

They added, “Staff with significant responsibility for research were determined through processes documented in codes of practice, and approved by the funding bodies. We know, in many cases, the processes developed to do this provided greater clarification on employment responsibilities, where previously there had not been clarity. We then looked at those decisions, on which staff had or did not have this significant responsibility, through audit to ensure they were made in line with the documented process.”