Coalition to reshape research evaluation must give national differences more attention, says Emanuel Kulczycki
Many research organisations in eastern Europe and former Soviet republics have joined the Coalition for Advancing Research Assessment, an international effort to reform how research is evaluated, including a move away from the crude use of metrics and rankings. There are 13 signatories from the Czech Republic, for example, eight from Hungary and 53 from Poland.
There is no one from eastern Europe, however, on Coara’s 11-member steering board. This risks leaving gaps in knowledge and understanding that will make the group’s aims harder to reach across roughly half of Europe.
Research is so international that it’s easy to think of it as a flat world, where everyone faces the same conditions and constraints. But national systems retain significant differences that often go overlooked in the enthusiasm for international reform efforts, which tend to be driven from western Europe.
Eastern Europe and the former Soviet republics have a long and distinctive tradition of research evaluation. As I discuss in my new book—The Evaluation Game: How Publication Metrics Shape Scholarly Communication—Russia has used publication counting as a means of evaluating research for two centuries, while Poland has used such methods for nearly a century. This is one of several reasons why, broadly speaking, in many parts of eastern Europe metrics are trusted more than experts.
The socialist era also deeply embedded the idea of the social function of science in these countries’ research systems, as a means of contributing to the economy and fulfilling Soviet ideals. Yet western European policymakers often treat societal impact as a recent discovery that they need to export to other, less up-to-date parts of the world.
On top of this, Soviet and socialist science management systems have left a legacy of centralised, national-level decision-making. Government-level incentives are still crucial in shaping academic advancements. These strong national legal frameworks create challenges for implementing pilot studies and adapting evaluation rules at the institutional level in order to diversify approaches to assessment.
Coara’s ambition to coordinate initiatives aimed at advancing research assessment is laudable. But I fear that, unless differences between national systems are better appreciated, international reform efforts are more likely to breed misunderstanding and conflict than comprehensive and inclusive change.
A collaborative approach
More effort needs to go into ensuring that reforms encompass both government-level initiatives and institutional-level changes. Coara has acknowledged the impediments posed by national regulations, but simply calling on national administrations to reform is unlikely to achieve much. A more proactive and collaborative approach is necessary.
This should include a systemic effort to end eastern Europe’s invisibility in research evaluation reforms; this is beyond the capabilities of individual scientists to ‘engage’. Such a change would bring in both crucial local knowledge and a more thorough understanding of the landscape of research evaluation.
An obvious first step would be for Coara to strive for greater diversity and inclusivity in its board, ensuring representation from all regions, including central and eastern Europe. This, along with the more active participation of signatories from the region, will ensure that the perspectives and experiences of these countries are brought into decision-making.
Second, there should be a stronger emphasis within the coalition on understanding and respecting national-level guidelines and policies, recognising their central place in research evaluation. This requires taking into account each country’s unique context.
Third and finally, effective reform efforts depend on fostering dialogue between international organisations, national administrations and institutions. A collaborative approach will enable the sharing of best practices and the identification of strategies that align with the specific needs of each region.
It is important to remember that cooperation also requires resources. If all potential solutions are produced by and promoted from the relatively wealthy institutions of western Europe, they are unlikely to travel well.
By recognising and valuing the contributions and perspectives of different countries, Coara can ensure its initiatives are truly inclusive and effective. Both those inside the coalition and those who have so far been left out must work to bridge gaps in understanding, promote dialogue and embrace diverse approaches and policy frameworks. Only through such concerted efforts can international reforms foster equitable and sustainable research evaluation practices worldwide.
Emanuel Kulczycki is head of the Scholarly Communication Research Group at Adam Mickiewicz University in Poznań, Poland
This article also appeared in Research Europe