Metrics as a measure of academic performance may be ready for implementation in a Research Excellence Framework sooner than previously believed, according to the managing director of research management at the publisher Elsevier.
Nick Fowler told a conference in London that while metrics were still a work in progress, some measures are already a reasonable predictor of performance.
This was in contrast to the interim findings of the Higher Education Funding Council for England’s review of the role of metrics in research assessment and management. The review will conclude that, in most cases, metrics are not yet ready to replace peer-review based judgements.
To illustrate his point, Fowler presented graphs generated with Elsevier software that showed a strong correlation between recipients of quality-related funding from the REF and those in the top 5 per cent of publications.
Fowler was speaking at a one-day conference of university leaders, managers and policymakers on 31 March. Elsevier had jointly organised the event with the Higher Education Policy Institute think tank. The conference was chaired by former universities and science minister David Willetts.
HEFCE’s metrics review is being carried out by a multi-disciplinary team led by James Wilsdon, professor of science and democracy at the University of Sussex.
Presenting interim findings from the year-long exercise, Wilsdon concluded that it is neither feasible nor wise to assess the quality or impact of research using quantitative indicators alone. He added that it would likely be another three cycles of the REF exercise before technology and academic community are ready for more concerted integration of metrics into the process.
Summarising his team’s findings Wilsdon said that metrics alone would be unable to capture those outputs that are not journal articles, which could disproportionately affect how humanities and more practice-based research are assessed. He also said that there were concerns around bias in relation to equality and diversity were more metrics to be used.
Research environment, he said, was one element of the REF that might be ready for more use of metrics. “No set of numbers, however broad, is likely to be able to capture the multifaceted and nuanced judgments on the quality of research outputs that the REF currently provides.”
The Wilsdon review’s interim findings also show that more involved use of metrics would not greatly reduce the cost of exercises such as the REF. This was in part because he said metrics could and should only be used to inform peer review rather than replace it.
One possible use for metrics, Fowler said, might be to select the information that peer review panels would then consider. He said that initial data analysis might be used to identify outliers, and then REF panellists might apply their insight to analysing those outliers. Fowler argued for example that, rather than encouraging gaming, well-selected metrics could drive positive behaviours. He also said that data sources on the humanities were becoming more complete.
Wilsdon said he agreed with Fowler that metrics have potential to complement human judgement. “People want improvements to be made,” he said, “but they want the system to be bedded down. The mood of the community, which we have weighed heavily, does not support radical changes at this point.”
Fowler said the UK was already ahead of the rest of the world in making use of metrics and in effect challenged funders and policymakers to help rather than hinder an important UK export industry. He said that the UK accounted for 4 per cent of the world’s researchers but 16 per cent of customers of Elsevier metrics services.
Wilsdon said the decision to incorporate more metrics into the REF was a complex one and “not to be done just because the algorithm is ready”.
The metrics review team’s final report is due to be published on 9 July.
Correction: The third paragraph has been corrected to read: "The review will conclude that, in most cases, metrics are not yet ready to replace peer-review based judgements". The word "replace" appears instead of "complement" in the original sentence. We apologise for this error which was introduced during editing.