Tests coming to market “without enough known on their effectiveness” says Royal Statistical Society president
An influential group of statisticians has called for an overhaul of data on diagnostic tests for Covid and future epidemics amid what they say is inadequate evidence and transparency of their effectiveness in real-life settings.
In a new report, the Royal Statistical Society’s Working Group on Diagnostic Tests called on the Medicines and Healthcare products Regulatory Agency to review and revise the national licensing process for diagnostic tests.
“Testing has been a key focus of many government’s strategies in fighting Covid, but the lack of statistical standards has caused issues, with tests coming to market without enough known on their effectiveness,” said RSS president Deborah Ashby, who co-chairs the working group.
“We urge regulators to take on board our recommendations to allow for more scrutiny of diagnostics more generally and for future pandemics.”
The group said there should be a “common statistical benchmark” for evaluating the performance of diagnostic tests.
It also said legislation should require tests to be evaluated in the settings where they will be used, and for that evidence to be independently scrutinised and publicly available.
Testing with ‘no data’
During the Covid-19 pandemic, the group said, “there have not been the necessary evaluations of tests for the different uses to which they are applied”. For example, they said that the initial claims of the sensitivity of lateral flow test were based on evaluations conducted in laboratory settings in patients with symptoms—but the tests were then used by non-healthcare professionals in people without symptoms in mass testing.
“Now they are being used in school-children and as a self-test, for which no data are available at all,” they added.
The group is also calling for more transparency around test performance, so people are aware of the likelihood of incorrect results, as false reassurance can lead to greater risk-taking and therefore increased transmission.
The group stressed that the assessment of the tests’ safety must include the potentially harmful consequences of false negatives and false positives.
“The proper assessment of the suitability of Covid-19 tests has been neglected,” said fellow co-chair Jon Deeks, a professor of biostatistics at the Institute of Applied Health Research at the University of Birmingham.
“Investment in well-designed studies evaluating tests in the real-world settings where they are used must become standard practice,” Deeks said. “We must learn from the mistakes made during the pandemic and put in place requirements for stronger science, better regulation and more transparency.”
Opportunity to ‘raise the bar’
Other experts have generally welcomed the findings.
Stuart Hogarth, a lecturer in sociology of science and technology at the University of Cambridge, described the innovation process for the development of new diagnostic tests as “poorly organised, under-resourced and lacking in scientific rigour”.
“Over the last two decades a succession of policy reports—often focused on genomics and personalised medicine—have highlighted a series of problems in diagnostic research, including studies that are statistically underpowered and/or prone to various types of bias, insufficient research on clinical outcomes, over-fitting of data in retrospective analyses and a lack of prospective controlled studies,” he said.
With the introduction of a stricter regulatory framework for diagnostics in the European Union, he said there was an opportunity to “raise the bar” in the UK, “and ensure greater public confidence in diagnostic tests, but we must ensure that the UK remains committed to this path”.
Joshua Moon, a research fellow in sustainability methods at the Science Policy Research Unit at the University of Sussex Business School, said the report’s recommendations made sense but did not go “far enough” on tackling the “complexity in communicating the nuance around these tests”.
“Testing isn’t just the test and the result but what people do with it,” he said.
Moon also questioned whether the government’s recent proposed Covid-19 testing regulation mechanism is “up to the task.”
In April the government announced proposals for “a validation regime that would require manufacturers to register via an online portal”, “submit data for a desk-based assessment” and “provide tests to undergo independent laboratory-based validation to ascertain their specificity, sensitivity and limits of detection using range of standardised samples”.
Responding to the RSS report, Graeme Tunbridge, the Medicines and Healthcare Products Regulatory Agency director of devices, said it was “currently considering changes to the framework for the regulation of diagnostics”.
“In the coming months, we’ll be consulting with the scientific community, healthcare professionals, manufacturers and the public to inform the design of this future framework.
“We would encourage all interested parties to respond to the public consultation.”