Environment statements map changing approaches to equality, say Matthew Inglis, Elizabeth Gadd and Elizabeth Stokoe
Between the 2014 and 2021 exercises, the Research Excellence Framework increased its focus on equality, diversity and inclusion. For example, the official guidance on preparing environment statements included notes on covering EDI beyond the ‘people’ section, presumably to show how action on EDI had been embedded in all areas of research and impact. Did this change how units of assessment, as the REF’s disciplinary categories are called, wrote their environment statements?
A text-mining analysis of the ‘people’ section of environment statements in REF 2014 found that discussions of what was then called E&D focused on gender (especially ‘women’ and ‘female’) more than other aspects of diversity. Higher marks in the assessment were associated with greater attention paid to E&D, realised through the use of keywords such as ‘equality’ or ‘diversity’.
We analysed all sections of 2021 environment statements to see which words appeared more frequently than in 2014. Some changes, like the appearance of ‘Covid’ and ‘Brexit’ in 2021, are easy to explain. More interesting are those that reflect the shifting expectations of the REF and changes to the broader context within which universities operate. In this way, we start to see how REF submissions form a historical record of UK research culture.
Of the 100 words that showed the biggest increase in usage between 2014 and 2021, 24 seemed to be EDI-related. For instance, the acronym ‘BAME’ appeared just once in REF 2014 environment statements but 3,741 times in REF 2021.
These terms fall into three clusters, illustrated in the figure below.
The first comprises words used to describe a people-centred culture: ‘our’, ‘we’, ‘wellbeing’, ‘colleagues’, ‘caring’ (listed from the most common). This dramatic increase in first-person pronouns and the active voice shows institutions seeking to stress their focus on people and inclusion.
The second cluster contains EDI-related terms: ‘EDI’, ‘diversity’, ‘inclusion’, ‘inclusive’, ‘equality’, ‘unconscious’, ‘bias’, ‘diverse’, ‘inclusivity’. As well as reflecting growing attention on EDI, this points to universities’ use of unconscious bias training, possibly because REF 2021 required all institutional decision-makers to be offered it.
The third cluster refers to characteristics: ‘BAME’, ‘female’, ‘gender’, ‘women’, ‘male’, ‘protected’, ‘race’, ‘LGBTQ’. While showing a broadening relative to 2014, it is interesting to see which words are not on this list, such as ‘age’, ‘class’, ‘disability’ or ‘neurodiversity’.
Clearly, the extent to which EDI policy and practice appeared in REF environment statements changed a lot between 2014 and 2021. In a recently published paper, we sought to gauge how this increase varied across statements and how it was reflected in assessment scores.
Topic modelling
Using topic modelling, a machine-learning technique to identify groups of words with similar meanings, we found eight topics that collectively predicted a large proportion—58.9 per cent—of the variance in units’ environment scores. One, which we called career development and EDI, was characterised by words such as ‘support’, ‘training’, ‘access’, ‘career’ and ‘diversity’. The amount of space devoted to these issues varied, from 0.4 per cent in the Royal Agricultural University’s agriculture, food and veterinary sciences unit of assessment to 39.9 per cent in Leeds Arts University’s statement for music, drama, dance, performing arts and screen studies.
We also found that the relationship between the space given to career development and EDI and environment grade point average was nonlinear, peaking at around 15 per cent of the statement and falling off for submissions with much more or less than this.
It’s easy to explain why submissions that neglected this topic tended to score poorly: they failed to follow the guidance to discuss EDI throughout submissions. Panellists may have concluded that this reflected a poor workplace for minoritised colleagues. But why might discussing these topics at length lead to a lower score?
One possibility is that, once a submission successfully demonstrated that career development and EDI was a matter of concern, additional discussion crowded out space that could have been better used to discuss other issues. A second possibility is that mentioning EDI too frequently conveyed a tick-box approach rather than an authentic embedding. Finally, an unusually lengthy discussion of EDI issues might give the impression of an unusually high number of issues in this area.
Given the decision to place an even greater focus on research culture in REF 2029, of which EDI is a key component, it will be interesting to see if and how this trend develops.
Matthew Inglis and Elizabeth Gadd are at Loughborough University. Elizabeth Stokoe is at the London School of Economics and Political Science
This article also appeared in Research Fortnight