One of the major concerns in biomedical research today is that many basic findings cannot be easily replicated by other labs. Might the literature be stuffed with unrepeatable results that were not sufficiently checked out before publication? Such doubts have increased in the past year, especially after scientists at Amgen and Bayer reported that they had been unable to reproduce the vast majority of ‘landmark’ papers describing promising approaches to treat disease.
How to raise standards? One answer has been to argue for more independent verification — but agencies are not sure how to fund such work, as it lacks the excitement of generating novel results.
Now a project set up expressly to replicate important research results has secured funding to check 50 high-profile studies, it announced today.
The Reproducibility Initiative, co-founded by former geneticist Elizabeth Iorns, says it has been awarded US$1.3 million from the Laura and John Arnold Foundation to validate 50 of the highest-impact cancer findings published between 2010 and 2012. They include 27 papers published in Nature, says Iorns, along with nine
three studies in Cell, three in Science and others from well-recognized journals such as Proceedings of the National Academy of Sciences, Cancer Cell and Nature Medicine. Update (29 October): a list of the papers involved has been posted at the Open Science Framework, here.
“The lack of reproducibility in cancer studies is a major obstacle in the development of viable therapies to cure cancer,” Iorns stated in a press release announcing the move. “The funding will be instrumental in not only verifying cancer studies, but also helping to institutionalize scientific replication.”
The work will be carried out through Science Exchange, a commercial online portal that matches scientists with experimental service providers, which Iorns co-founded. She says that independent scientists will tell authors when they are trying to replicate their work, and will ask them to check the planned methods and request specific materials. The study results will be published piecemeal in PLoS ONE; Iorns expects that they will all be done by September 2014.
When Nature last wrote about whether federal agencies such as the US National Institutes of Health might require independent validation of studies, not everyone was happy with the idea. “It is unbelievably difficult to reproduce cutting-edge science,” warned Peter Sorger, a systems biologist at Harvard Medical School in Boston, Massachusetts.
But William Gunn, co-director of the Reproducibility Initiative, said in the press release today: “This project is key to solving an issue that has plagued scientific research for years.”
The Arnold Foundation, based in Houston, Texas, has a history of supporting efforts for open science and research integrity. Earlier this year it gave away $5.25 million to help establish the Center for Open Science, based in Charlottesville, Virginia, which hopes to encourage the openness, integrity and reproducibility of science research. With the funding today, the Reproducibility Initiative is also announcing its collaboration with the similarly named Reproducibility Project, a crowd-sourced effort by scientists to reproduce many published studies in psychological research (see ‘Replication studies: Bad copy’). The Center for Open Science will administer the funding for the project.