Achieving gender balance in the evaluation of research proposals can be tricky. Even though most research councils have procedures in place to ensure that applications from men and women are considered on an equal basis, the fact that bias against women may happen unintentionally makes it a difficult problem to overcome.
In search of a solution, the Swedish research council Vetenskapsrådet started doing observation studies in 2008 and a gender mainstreaming exercise in 2014. The council has examined every step of the grant evaluation process to understand how unintentional gender bias happens and come up with ways to prevent it. In doing so, it has moved away from the reactive gender policies of its counterparts in other Nordic countries, towards a fundamental rethink of evaluation.
On 6 March, Vetenskapsrådet published the results of an analysis of eight of its evaluation panels. This looked at how evaluators were recruited and instructed and how applications were reviewed, and assessed the finest details of panel conversations to see how they played out and which evaluators’ opinions carried the most weight. It also evaluated the effectiveness of pre-allocated seating arrangements for panel meetings, aimed at improving the group dynamic.
Interestingly, the council concluded that the informal parts of evaluation meetings, such as spontaneous discussions and chats at coffee breaks, which evaluators might enjoy most, have a significant negative effect on the success of women. Because Sweden is such a small country, there is the age-old problem of the old boy network, the council found—leading to researchers with established reputations being picked again and again. These researchers are often men.
It also found that evaluators tended to look at the final numbers of applicants from each sex to determine whether there had been gender bias in the process, rather than considering how men and women were treated differently throughout the process. If a woman has done a lot of research in a specific area, her experience is likely to be judged as too narrow, whereas for a man this is seen as indicating that he must have in-depth knowledge of the subject, says Kerstin Alnebratt, the director of the Swedish Secretariat for Gender Research at the University of Gothenburg, which helped coordinate the mainstreaming exercise.
“Everyone says that they don’t deal with men and women differently, but when you scratch that surface you’ll see that they do,” says Alnebratt.
On a positive note, Vetenskapsrådet found that efforts to formalise the evaluation process by introducing seating arrangements and enhancing the role of the person chairing the panel was effective. Alternating male and female panel members around the table and assigning fixed topics for discussion prevented men from dominating proceedings, the council found, leading to better outcomes for female applicants.
In its conclusions, Vetenskapsrådet said that such procedures should become a set part of its evaluation guidelines. Despite acknowledging that firming up procedures and cutting down on informal chats between evaluators could be seen as “blunt and bureaucratic”, the council said that these measures would increase its ability to fund the best research. “We see that it would reduce the risk of informal and non-defined criteria becoming significant to the outcome,” the report says.
Perhaps other research councils should look just as closely at their procedures, to see whether similar changes might be appropriate. “Deconstructing the peer-review process is important for every research council if you really want to find the highest-quality research,” says Alnebratt. “You have to keep doing this work over and over again, and realise that no-one is immune against gender bias in peer review.”
This article also appeared in Research Europe