Research consultant

Less than a third of cancer research results are reproducible

The researchers used a combination of automated text analysis and the “scientific robot” Eve to semi-automate the process of reproducing research results. The problem of lack of reproducibility is one of the greatest crises facing modern science.

The researchers, led by the University of Cambridge, analyzed more than 12,000 research papers on the biology of breast cancer cells. After reducing the set to 74 articles of great scientific interest, less than a third – 22 articles – turned out to be reproducible. In two cases, Eve was able to make fortuitous discoveries.

The resultsreported in the review Royal society interfacedemonstrate that it is possible to use robotics and artificial intelligence to help solve the reproducibility crisis.

A successful experiment is one where another scientist, in a different laboratory under similar conditions, can achieve the same result. But more than 70% of researchers have tried in vain to reproduce the experiments of another scientist, and more than half have failed to reproduce some of their own experiments: this is the reproducibility crisis.

“Good science relies on the reproducibility of results: otherwise the results are virtually meaningless,” said Professor Ross King of Cambridge’s Department of Chemical Engineering and Biotechnology, who led the research. “This is particularly critical in biomedicine: if I am a patient and read about a promising new potential treatment, but the results are not reproducible, how am I supposed to know what to believe? The result could be that people lose faith in science.

Several years ago, King developed the science robot Eve, a computer/robotic system that uses artificial intelligence (AI) techniques to perform scientific experiments.

“One of the big advantages of using machines to do science is that they are more precise and record details more accurately than a human can,” King said. “That makes them well suited to the job of trying to replicate scientific results.”

As part of a DARPA-funded project, King and his colleagues from the UK, US and Sweden have designed an experiment that uses a combination of AI and robotics to help solve the reproducibility crisis. , making computers read and understand scientific papers, and getting Eve to attempt to replicate the experiments.

For the current paper, the team focused on cancer research. “The cancer literature is huge, but no one ever does the same thing twice, which makes reproducibility a huge issue,” King said. “Given the huge sums spent on cancer research and the huge number of people affected by cancer worldwide, this is an area where we urgently need to improve reproducibility.”

From an initial set of more than 12,000 published scientific papers, the researchers used automated text mining techniques to extract statements related to altered gene expression in response to breast cancer drug treatment. . From this set, 74 articles were selected.

Two different human teams used Eve and two breast cancer cell lines and attempted to replicate all 74 results. Statistically significant evidence of repeatability was found for 43 articles, meaning that the results were reproducible under identical conditions; and significant evidence of reproducibility or robustness was found in 22 papers, meaning the results were reproducible by different scientists under similar conditions. In two cases, the automation made incidental discoveries.

While only 22 of 74 items were found to be reproducible in this experiment, the researchers say that doesn’t mean the remaining items aren’t scientifically reproducible or robust. “There are many reasons why a particular result may not be reproducible in another lab,” King said. “Cell lines can sometimes change their behavior in different labs under different conditions, for example. The biggest difference we found was that it matters who is experiencing it, because everyone is different.

King says this work shows that automated and semi-automated techniques could be an important tool to help solve the reproducibility crisis, and that reproducibility should become a standard part of the scientific process.

“It’s quite shocking how much of an issue reproducibility is in science, and it’s going to require a complete overhaul of the way a lot of science is done,” King said. “We believe machines have a key role to play in helping to fix it.”

The research was also funded by the Engineering and Physical Sciences Research Council (EPSRC), part of UK Research and Innovation (UKRI).

– This press release was originally published on the University of Cambridge website