Problems with reproducing the findings of biomedical research began to receive widespread attention in 2012, when the pharmaceutical company Amgen announced that it had been unable to reproduce the findings of 47 out of 53 “landmark” cancer papers.
Similar studies followed, said Anton Ussi, operations director at the European Infrastructure for Translational Medicine in the Netherlands, during a session on 27 July. “None have shown that 50 per cent or more of the findings tested were reproducible.”
The previous day, the journal Scientific Reports published a paper showing “striking” variation in a single batch of MCF-7 cells, which have been used as a model for cancer research for over 40 years, contributing to almost 23,000 papers. The authors concluded that standard cell authentication processes might not detect variability “that could fundamentally compromise reproducibility”.
Speaking in Manchester, the paper’s senior author Thomas Hartung—a toxicologist at Johns Hopkins Bloomberg School of Public Health in the United States and University of Konstanz in Germany—said that 92 per cent of clinical trials failed to produce effective treatments, partly because preclinical researchers are working with tools that are far from perfect. “A lot has to do with non-standardisation of methods and a lack of quality control,” he said.
On the bright side, newer disease models based on patients’ own tissues are due to deliver a “paradigm change” in the quality assurance of cells, Hartung said. An international collaboration on Good Cell Culture Practice, to bring together funders and research organisations, was formally launched by Hartung and his colleagues during the ESOF conference.
The session that Ussi chaired covered many other potential causes of non-reproducibility, including the pressure on researchers to publish, a lack of statistical knowledge, the inaccessibility of raw data for re-testing and a failure to sufficiently flag up flawed papers.
Justin Bryans, director of drug discovery at the technology arm of the UK’s Medical Research Council, said that between 60 and 70 per cent of his team’s attempts to advance basic research “fail because the hypothesis is wrong”.
He said that funders have “a huge role to play” in improving biomedical reproducibility. “They have the ability to stipulate what academics should do in terms of quality control. We need to change the behaviour of academics,” he said.
Frances Rawle, head of governance and policy at the MRC, agreed that funders could help improve peer review, emphasise the importance of methodology and pay for statistical support and good data management. However, she said that the MRC was not considering funding replication studies at the moment. Rawle said that she was “having trouble convincing her colleagues that, given the pressures on other funding, it would be a good use of their money”.
This article also appeared in the ESOF 2016 Special