Go back

Assessment reform ‘still needs metrics’

Image: Grace Gay for Research Professional News

Earma 2023: A qualitative approach alone is not enough, university leader says

Attempts to reform research assessment will not be able to abandon metrics entirely as there is still a need to assess work on nuanced but defined scales, the annual conference of the European Association of Research Managers and Administrators has heard.

There is currently an EU-wide push to reform research assessment, set up as a response to growing disquiet in the research world about the use of metrics to assess institutions and academics.

The EU coordinated the Agreement on Reforming Research Assessment, which launched last year and has since gained signatures from more than 500 research organisations. It calls for assessment to be primarily based on qualitative evaluation and for the “inappropriate use of metrics” to be abandoned.

But Ladislav Krištoufek, vice-rector for research at Charles University, told conference delegates in his institution’s city of Prague on 25 April that numbers are still needed.

“We need to acknowledge that the qualitative approach to research assessment is super important…but, on the other hand, if you need to make managerial and budgetary decisions, that is not enough,” he said. “You need something that scales.”

‘A’ for effort

Metrics are needed in research assessment as “letters don’t scale”, said Krištoufek. For example, giving institutions grades such as A and B might not be nuanced enough to help national funders and governments decide how to distribute money between them.

Assessment of an institution’s research performance and outputs is often linked to the funding they receive. “At the national level, you need a system to give [institutions] money,” said Krištoufek. While numbers are needed, this does not have to necessarily be a publishing metric, he said.

He acknowledged that a call to keep numbers in research assessment raises the question of what metric or metrics should be used and how they could meet the ideas in the reform agreement—which also includes recognising the diversity of research roles and avoiding using international rankings in assessment.

At his own university, Krištoufek said there had been a discussion recently on how to approach the use of metrics in assessment in a just and equal way, which he said is “always difficult as nothing is just in reality”.

He added that another challenge with the Agreement on Reforming Research Assessment is that implementing it will be “super costly” and there are not enough people to carry out its ambitions.

Financial support

The Coalition for Advancing Research Assessment has been set up to implement the agreement and has so far amassed over 400 members.

Stephane Berghmans, director of research and innovation at the European University Association, which helped draft the agreement, said at the conference that research management would have a “very big role” to play in the push to reform assessment in academia.

He echoed Krištoufek’s funding concerns, saying that money is needed to support the reforms, which should come from both the EU and national funders.

“If new rules are imposed [to implement research assessment reforms]…we need to ensure that there is financial support so that we have the staff both in terms of number but also the right skills—that is going to be fundamental.”

Research Professional News is media partner for the Earma 2023 conference in Prague. Read all the coverage here.