Go back

Simpler REF remains a mirage


Arma 2021: ideals behind reforms are not playing out in operational reality, says Anna Grey

In his review of the 2014 Research Excellence Framework, Nicholas Stern highlighted concerns with the pressure on academic staff to produce outputs; potential game-playing by institutions over which staff were returned; and the poaching of academics close to the submission date. 

To address these issues, he proposed breaking the link between authors and papers, with institutions submitting pooled outputs. Work from former staff would be eligible, but those moving jobs would leave their outputs behind—making poaching pointless—and each individual could submit anywhere between one and five outputs. 

This flexibility was also intended to save people from detailing the personal circumstances that prevented them from producing the expected number of outputs. As a final benefit, it should have resulted in a simpler, less burdensome REF.

As REF2021 works its way through the assessment phase, however, it’s becoming evident that institutional REF managers have found themselves squeezed between Stern’s theoretical ideals and cold, hard operational reality. 

Breaking the link between people and outputs submitted is a sound proposal. But in reality there is, not surprisingly, a strong link between academics and their outputs. 

To fill the pool, academics must highlight their best work, which then needs to be internally graded. This still feels like a personal assessment—and universities are unlikely to move away from managing research based on individual performance. Identifying and grading outputs from ex-staff also creates a new job.

The next issue was the concern that a small number of stars would dominate the output pools, and so be given all the resources. To avoid this, each academic submitted had to contribute at least one output and no more than five. 

However, academics in a department often co-publish, whereas outputs could normally only be submitted once. Trying to optimise the return so that the best outputs were included while meeting the criteria for submission was complex. For individual academics, it created a new perceived pressure of not only being graded, but also around how many of their outputs would be chosen. 

The final element in Stern’s reforms was the vexed issue of personal circumstances. The last REF had a complex model for reducing the number of outputs required from staff with, for example, health issues or caring responsibilities. 

This put REF managers in the uncomfortable position of having to ask for very personal information and then applying a formula to work out the reduction in outputs. They quite rightly pushed back, and it was something the move to the pool of outputs should have resolved. 

Once again, however, trying to break the link between individuals and outputs is psychologically difficult. And if an individual has personal circumstances, it only seems fair that the REF takes this into account. 

Creating more burden

The compromise was a complex system of reductions applied to units rather than individuals. But units could only apply to reduce their output pool if a significant number of staff came forward with circumstances, meaning that staff still had to divulge highly personal information.

Perhaps inevitably, then, significantly changing the REF created a significant burden, as staff tried to work out the implications of the changes and how to implement them. This put REF managers in a bind. 

There was not enough time to work through the operational aspects of Stern’s proposals, and a desperate need to get the final rules so that systems could be put in place to manage the process. When the REF changes, the sector as a whole also needs time to change culture and work through what the changes mean.

Another round of review and reform will follow REF2021. We need to accept that any significant changes that result will create an immediate increase in workload because the intellectual and theoretical framework has to be put into operation. There should be a balance between tweaking the current process, given much of the implementation work has been done, and something more radical, where there is an aim of reduced work, but no guarantee.

This is not to say change should not happen, rather that its effects on the ground should not be ignored. Any panel charged with suggesting changes to the REF needs to apply a healthy dose of pragmatism to its theoretical goals; this would be helped enormously if such panels included an experienced REF manager so that these implications could be understood at the start rather than flagged up afterwards. 

But finally, perhaps the lesson is not to promise to reduce the burden of the REF when that is clearly not possible.

Anna Grey is research strategy and policy manager and deputy head of research services at the University of York. She is speaking at the annual conference of the Association of Research Managers and Administrators on 6 October

This article also appeared in Research Fortnight