Consortia struggle with benchmarks and quality
A shortage of relevant data has hampered the quality of the second set of science and innovation audits, according to those who compiled them.
The audits, announced in 2015, aim to inform the government about R&D opportunities in regions across the UK, and to promote such investments outside London and the south-east of England. The first five reports were published in November 2016 and a further eight were published by the Department for Business, Energy and Industrial Strategy (BEIS) on 21 September.
The authors of some of the reports have told Research Fortnight that they encountered problems related to the quality of data available.
Technopolis, the consultancy that provided analytical support for the audits, supplied datasets that offered “very limited insight” and were “quite out of date”, said John Whaling, who leads innovation and strategic investment at the Liverpool City Region Local Enterprise Partnership. “The data didn’t provide the step change in insight for which we were hoping,” he said. Most of the information was econometric, while Whaling said he would have liked datasets on real-time business activity and industry investment.
Mark Mortimer, a director in the research and enterprise office at the University of York who contributed to a report on the bioeconomy of the north of England region, agreed. “The data provided by Technopolis was the data they had, not necessarily the data that answered the questions we had,” he said.
The consortia that compiled the audits also experienced difficulties benchmarking how their regions performed compared with other places. “This is an issue we raised at the beginning, the middle and the end of the process, and something that BEIS absolutely acknowledged was a challenge,” Whaling said.
Paul Simmonds, UK managing director at Technopolis, admitted that there was limited available data—in particular for emerging technologies, nascent industries and smaller businesses—and that this had led to “some frustration” among report authors.
Ahead of the third wave of audits, which BEIS said it would announce in the coming months, Technopolis is looking to provide more data on investment and trade—two areas frequently requested—to create a more open and interactive data facility, and to explore how to better analyse the data.
“For a lot of the people who were responsible for coordinating these audits and writing the reports, this kind of data mining is not necessarily their particular strength,” Simmonds said.
Problems with data have had a damaging effect on the quality of the audits, said Kieron Flanagan, senior lecturer of science and innovation policy at the University of Manchester. He described most of the documents as “glossy prospectuses” rather than a deep analysis of strengths, weaknesses, opportunities and threats.
The audits proposed a number of government investments, such as setting up a biomanufacturing park in the north of England and a node of the Compound Semiconductor Applications Catapult or the Energy Systems Catapult in Scotland’s central belt. But Flanagan said the package of audits could prevent the government from deciding what to prioritise, as all the regions tried to make themselves look good.
Mortimer admitted that each consortium had tried to present its region in the best light. “You could say that there is an inherent bias in the process,” he said.
However, he added that the exercise allowed each consortium to build a more realistic picture of its strengths. “We thought we were very good at synthetic biology, for instance, but the evidence showed that although we were as good as everybody else in the country, we were not necessarily world leaders.”
The eight consortia that compiled the audits are: Bioeconomy of the North of England; East of England; Glasgow Economic Leadership; Innovation South; Leeds City Region; Liverpool City Region Plus; Offshore Renewable Energy; and the Oxfordshire Transformative Technologies Alliance.
This article also appeared in Research Fortnight and Research Europe