Most scientific work isn’t reproducible. Andy Tay explains why that’s a problem.
The call for reproducibility has never been stronger in the history of science. Since two major pharmaceutical companies, Amgen and Bayer, reported in 2011/12 that their scientists were unable to replicate 80-90% of the findings in landmark papers, scientific news outlets have caught up on the issue. Their reports have catalyzed conversations among stakeholders (policy makers, funding agencies and scientists) to improve reproducibility in science.
There are a lot of reasons why reproducibility is so important, and why Amgen and Bayer’s results caused such controversy. I’ll start at the individual level.
Individual
A survey among elite senior scientists revealed that they feel traits like honesty and curiosity are the ‘keys to scientific excellence.’ These are, presumably, the values they try to identify in PhD applicants to groom as scientific leaders. So it’s reasonable to expect trained scientists to have these traits in spades.
However, the stress from competition to graduate, obtain tenured positions, publish papers, or secure grants can tempt scientists to publish sloppy results quickly, in large numbers and (if they can) in high-impact journals. The outcome is that scientists trapped in this rat race may eventually dismiss that initial, honest motivation for starting off in a research career in the first place.
Organizational
Poor reproducibility in science often leads to financial losses in the form of research materials and manpower to all institutions in the business of science. Beyond these direct economic costs, there’s also damage inflicted on organizations’ reputations – built over years of hard work. A university notorious for erroneous science can lose top scientists who value a transparent research culture. Furthermore, reviewers may be cautious about rewarding grants to institutions or laboratories with lax reproducibility policies.
At the organizational level, enforcing reproducibility in research work goes beyond immediate financial losses as it is also paramount in building and maintaining a culture of integrity to retain and attract top students, scientists, collaborators, and even private and governmental funding.
Societal
Another survey, this time conducted by the American Association for the Advancement of Science (AAAS), found that an overwhelming majority of U.S. citizens believe that science has improved their quality of life and that public investments in science generally pay off.
For science to meet the expectation of improving the well-being of mankind at the fastest pace possible, each piece of scientific finding must be reliable. This is especially true as research becomes progressively inter-disciplinary and reproducible results must be gathered from a variety of fields – scientists who aren’t experts in a particular field must trust others’ results.
At the societal level, ensuring reproducibility in science helps to maintain the public’s faith in science and to ensure continual support for scientists who wish to utilize science to elevate societal well-being.
The economic loss in supporting irreproducible science is obvious. To motivate ethical research behaviors from scientists, education should start from in-the-lab discussions where individuals can openly share their worries and receive advice about graduation, publishing and funding. Such conversations can also remind scientists of their initial motivations for choosing a career in research, and roles as advocates and pursuers of truths in their institutions and societies.
Andy is a PhD student in the bioengineering department of the University of California, Los Angeles.
His research focuses on the evolution of magnetotactic bacteria and biophysics of neurons. In his free time, Andy enjoys using the gym and writing.
You can find Andy on Google Scholar or LinkedIn
Suggested posts
Reproducibility: Seek out stronger science
Turning point: Reproducibility pro
#Scidata15: Make the most of your research: Publish better data
Report this comment
I wouldn’t go as far as saying that “Most scientific work isn’t reproducible”. However, I would agree that much scientific work isn’t significant :)
More importantly, scientific work can only be reproducible if the samples sizes are large enough AND/OR if the experimental details are documented in sufficient detail — which is rarely the case. We could not even reproduce our own data when small experimental details were changed, but as long as they are kept identical they were highly reproducible (Chen et al. 2010, Nature Methods7 (9): 667).