International meeting will explore common approaches to responsible evaluation, say Dorsamy Pillay and David Sweeney
Before the Covid-19 pandemic, Research England and the National Research Foundation of South Africa were due to co-host a conference on responsible research assessment, as a satellite event to the annual meeting of the Global Research Council. The event was scheduled for Durban, South Africa, in May 2020.
As it grew apparent that the GRC conference could not take place on location, it became even more obvious that a global discussion on this subject was essential. As well as highlighting researchers’ societal impact, Covid-19 has sharpened the challenges faced by funders and researchers.
Researchers worldwide have been pressured to respond to the medical and social impacts of the pandemic. Many have had their work disrupted. Lockdown has had a disproportionate effect on female researchers. All this, and more, has highlighted the need for robust, fair and transparent models of research assessment.
In this spirit, and in recognition of the need for collaboration across borders during the pandemic, we moved the conference online, where it will run from 23-27 November; the plenary is open to all. The virtual event is a unique opportunity for an international discussion on issues that individual funders have grappled with for years.
Research assessment is an important driver of researchers’ behaviour. It is crucial that assessment criteria and processes incentivise and catalyse work that is trustworthy, rigorous and ethical.
Assessment can have a positive impact: funders’ requirements around open-access publishing have done much to advance open research. But it can also incentivise bad behaviour. For example, evaluation based solely on publications can pressure researchers to publish in high-impact journals. This makes them less likely to publish negative results and, in extreme cases, pushes them towards misconduct.
Research integrity
Competition in research funding is necessary, but it must allow a diverse range of researchers to thrive. Systems and models that value, or are perceived to value, prestige, novelty and a narrow range of outputs over excellence, impact, rigour and transparency risk promoting practices that undermine research integrity.
These issues go deeper than individual funding calls or national assessment exercises. A collective, global effort is required to bring about culture change.
The disruption caused by Covid-19 was abrupt and unforeseeable, but we were already working in a fast-changing world. Advances in technology, for example, are shaping how research is produced, disseminated and evaluated.
Developments in machine learning mean that peer review by artificial intelligence is no longer the province of (academic) science fiction. Some publishers have trialled using AI tools to automate their processes, from selecting reviewers to checking for plagiarism. Another AI tool claims to give a more nuanced picture of an article’s influence than citations alone can provide.
Before we embrace new technologies to assess research, we must understand what it means to assess research responsibly. If we don’t know what good practice looks like now—and if we don’t compare new approaches with the benefits of established systems such as peer review—how can we make the most of AI?
Funders are part of a complex international ecosystem. We have a shared responsibility and must work together—and in partnership with publishers, policymakers, compilers of university rankings, research organisations and researchers themselves—to support a healthy, dynamic, diverse research culture that can respond to the world’s greatest challenges.
The conference will build on the GRC’s previous work on merit review, research integrity and interdisciplinary research, and on momentum building across the globe. Although frameworks such as the San Francisco Declaration on Research Assessment are well embedded in many countries, not all regions are starting from the same point.
Ahead of the conference, we asked the Research on Research Institute in London, in partnership with Leiden University, Dora and the GRC, to produce a working paper setting out the state of play, to be published on 18 November. This builds on a recent study of research assessment practices by Science Europe and an online survey of GRC members.
The conference gives public funders worldwide their first opportunity to discuss common aims, and to begin a discussion with other stakeholders. It offers an opportunity to develop a shared understanding of responsible research assessment, learn from each other and build consensus on promoting and supporting positive research cultures.
Dorsamy (Gansen) Pillay is deputy chief executive officer of the National Research Foundation of South Africa. David Sweeney is executive chair of Research England