Author Archives: Elizabeth Hawkins

A dedicated home for computational science

Posted on by

This guest blog comes from Fernando Chirigati, Chief Editor of Nature Computational Science.

 

Recent advances in computer technology, be it in hardware or in software, have revolutionized the way researchers do science: problems that are too complex for human or analytical solutions are now easier to address; problems that would take years to solve can now be unraveled in days, hours, or even seconds. The use and development of advanced computing capabilities to analyse and solve scientific problems, also known as computational science, has undoubtedly played a key role in transformational scientific breakthroughs of our last century, making progress possible in many different disciplines. 

Providing a multidisciplinary forum for computational science

Given the growing interest in and throughput of computational science research, Nature Research is launching Nature Computational Science, a new journal that will provide a home for exceptional work being done in this field. The primary focus of the journal will be on publishing research related to the development and use of computational techniques and mathematical models, as well as on their application to address complex problems across a range of scientific disciplines, including, but not limited to, bioinformatics, cheminformatics, geoinformatics, computational physics and cosmology, materials science, and urban science.

The multidisciplinary nature of computational science will be one of the journal’s strengths, creating an opportunity to bring together researchers from different scientific disciplines who can learn from each other. Nature Computational Science will offer a high-quality, curated forum for stimulating such a multidisciplinary environment, where all of the computational work across these different fields can be found in a single, dedicated place, thus fostering discussion and collaboration. 

We welcome submissions on both fundamental and applied research, from groundbreaking algorithms and frameworks that notably help to advance scientific research, to methodologies that use computing capabilities in novel ways to find new insights and solve challenging real-world problems. We also encourage researchers and developers to submit software systems that have a significant impact on science and that help scientists to experience aha moments.

Like the other Nature titles, we have a dedicated team of in-house manuscript editors, production editors, and editorial assistants in place. Our editors will engage with research communities in academia and industry to provide exceptional author and reviewer service. The journal will uphold high-quality editorial and publishing policies.

By commissioning Comments, Reviews, Perspectives, and News & Views articles, Nature Computational Science will provide accurate, up-to-date background on a variety of relevant topics, as well as encourage discussion on timely matters. Given our multidisciplinary community, we will highlight content that is of interest to a broad audience.

Nature Computational Science will be published online-only, and for submissions from January 2021, the journal will be a Transformative Journal, meaning that Plan S-funded authors, as well as others wishing to publish open access, can submit their primary research to these journals safe in the knowledge that they will be complying with their funders’ requirements.

Fostering reproducible research

Many of the problems that computational science tackles today affect millions of people, which makes it integral to ensure that the complex computational analyses result in conclusions that are trustworthy and actionable. Nature Computational Science will champion the reproducibility of scientific outcomes, ensuring that articles meet the highest standards of reproducibility and transparency in reporting.

Our editors will work with authors and reviewers to pay close attention to the peer review and sharing of data and code. Code will be peer reviewed when central to the manuscript. As other Nature journals, we will be offering authors the option to use container-based platforms to facilitate code peer review and publication (see details here). Nature-branded journals strongly encourage researchers to use open repositories to share their code and data upon publication and to choose a license approved by the Open Source Initiative.

Recognizing the important role of preprint posting in the process of open scientific discourse, Nature Computational Science will offer authors the option to add a link from the published paper to its corresponding preprint, thus ensuring that the links are visible to all readers. Authors will be able to use established preprint servers such as arXiv and bioRxiv.

Launching in January 2021

Nature Computational Science will launch in January 2021, and it is now open for submissions. Find out more on our website. Any questions can be directed to computationalscience@nature.com 

Journals test the Materials Design Analysis Reporting (MDAR) checklist

Posted on by

Reproduced from summary presentation, available at https://osf.io/znq64 

This guest blog comes from Sowmya Swaminathan, Head of Editorial Policy and Research Integrity at Nature Research.

We are pleased to share results from a pilot with 13 journals that tested the Materials Design Analysis Reporting (MDAR) checklist, a minimum standards reporting checklist for the life sciences.

The MDAR framework, a minimum standards reporting framework for life sciences, was designed to provide a harmonizing principle for reporting requirements currently in use at various journals. It is meant to be flexible to adapt to various journal policies, and provides two levels of reporting stringency, a minimum recommendation and a best practice recommendation. The checklist was designed as an optional instrument to help adoption of this reporting framework. A statement of task can be found here.

We are very grateful to the 13 participating journals and platforms – BMC Microbiology, Ecology & Evolution, eLife, EMBO journals, Epigenetics, F1000R, Molecular Cancer Therapeutics, Microbiology Open, PeerJ, PLOS Biology, PNAS, Science, Scientific Reports – for testing the MDAR checklist.

The pilot had two main goals: first, to understand whether the checklist was accessible and useful for authors and editors to help comply with journal policy and second, to understand whether the elements within the checklist are clearly conveyed so as to help fulfil policy expectations. In total, 211 authors across participating journals tested the checklist and provided their feedback. Participating journal teams screened 289 manuscripts using the checklist and 89 of these manuscripts were subject to a dual-assessment by independent reviewers, which allowed us to determine inter-assessor agreement, and thus clarity of specific items on the checklist.

We are encouraged to find that about 80% of authors and editors found the checklist useful to different degrees and that the majority of participating editors reported only a small increase in manuscript processing time as a result of using the checklist. While participating authors and editors did not identify major gaps in the requirements covered in the checklist, the feedback from authors and editors and the inter-assessor agreement results have given us a better understanding of areas in the checklist and elaboration document where the language is unclear and needs to be improved.

We are making the draft MDAR Framework, MDAR Checklist and MDAR Elaboration document and the pilot datasets available here. This work was also recently presented at the NASEM workshop on Enhancing Scientific Reproducibility through Transparent Reporting (slides available here).

We are currently gathering feedback on the MDAR framework, checklist and elaboration document from a broad group of about 40-50 experts on transparency and reproducibility. Based on the feedback from the pilot and the expert input, we anticipate revising all three MDAR outputs by the end of 2019.

We are sharing this update on the work of the Minimum Standards Working Group through coordinated posts on member platforms. If you would like more information about our work and progress, please contact Veronique Kiermer and Sowmya Swaminathan.

On behalf of the “minimal standards” working group:
Karen Chambers (Wiley)
Andy Collings (eLife)
Chris Graf (Wiley)
Veronique Kiermer (Public Library of Science; vkiermer@plos.org)
David Mellor (Center for Open Science)
Malcolm Macleod (University of Edinburgh)
Sowmya Swaminathan (Nature Research/Springer Nature; s.swaminathan@us.nature.com)
Deborah Sweet (Cell Press/Elsevier)
Valda Vinson (Science/AAAS)

What we have learnt testing container-platforms for peer review and publication of code

Posted on by

A year ago now, we launched a trial to test the use of cloud-container platforms for peer review and publication of code at several Nature journals. The trial phase of this initiative is now officially over, and we would like to share the experience and outcomes, and provide an overview of what comes next.

What problem are we trying to solve?

Our guiding principle is that when new code is central to the main claims made in the paper, it is imperative that the code meets the same quality and reproducibility standards as the paper itself. This means the code needs to be properly documented, evaluated by experts so that it is functional (ie peer reviewed), and permanently identified and accessible at the time of publication to ensure the reproducibility of the results (these same principles apply to other research objects like data and protocols, but those were not the focus in this particular trial).

Nature Methods adopted the practice of ‘peer reviewing code’ for software papers in 2007 (editorial). Under this practice, editors require authors to submit the source code, a test dataset and details of installation, and ask the reviewers to install and test-run the code during peer review. This form of peer review is highly time consuming for authors, editors and reviewers, but it is also necessary. It is not uncommon for reviewers to point to basic flaws in the instructions or files that could render the code completely unusable.

Over a year ago, we partnered with Code Ocean, a Docker-based platform that allows authors to deposit code and data and enables users to run the code on the cloud with the set parameters to reproduce the results, or execute the code with new input values.  Together we developed a set of workflows and basic functionality of the platform that enables authors to upload the code and data associated with their submission, and reviewers to access the platform anonymously during peer review (see figure, reproduced from reference 1).

The trial was meant to evaluate if such a platform would provide:

  • A service to authors by assisting them in depositing the code and data and compiling them in an open, executable-based platform.
  • A service to reviewers by making code peer review easier (as easy as clicking a button). Reviewers can evaluate the code in the cloud using computing time that we provide as a publisher, not their own.
  • A service to readers by providing the code associated with the paper in a way that is properly identified, documented and supplied in a publically accessible platform that allows running, reusing and repurposing the code.

The trial was optional for authors at the three participating journals (Nature Methods, Nature Biotechnology and Nature Machine Intelligence) and we tracked feedback from authors and reviewers, author opt in rates and user-engagement metrics.

What have we learnt? Results!

Over 95 papers have now participated in the trial and more than 20 published papers are providing open, verified, properly documented and cited code using the technology.

Despite the additional work that authors need to do upfront when they sign up to the trial, we’ve seen large author uptake, with 54% of authors across all journals opting in to participate. Importantly, our reviewers actively engage with the platform. Capsules have received an average of 34 views via the private links provided to the reviewers. Approximately half of the reviewers signed up and duplicated the capsules, a requirement for running the code. Each reviewer that signs up runs the code 1.3 times, on average.  Importantly, peer review of code in this manner has surfaced problems with some manuscripts that would have led to the ‘irreproducibility’ of the code and the results.

Upon publication, we provide the links to access the code and data in a ‘Code Availability’ statement of the paper, which is provided openly to all readers regardless of access status.

We are looking at ways to improve the workflows and experience by providing better information, an easier workflow for editors and authors and better ways of surfacing code that is shared openly and peer reviewed through the use of badges.

We have been very pleased to see that the high standards we are applying to ensure open science and reproducibility of code in our papers has been noted, as we’ve received very positive feedback about the initiative from authors, reviewers and the science community. You can read more about the initiative and the results in the below editorials and in the Science Editor piece that we recently published.

Science Editor: Three approaches to support reproducible research

Nature Biotechnology: ‘Changing coding culture’

Nature Machine Intelligence: ‘Sharing high expectations’

Nature Methods: ‘Easing the burden of code review’

What’s next?

Given the positive effects we’ve seen so far, we will continue the current practice at the journals. We also want to learn how the workflow would scale and to test it on more scientific disciplines so we have added Nature, Nature Protocols and BMC Bioinformatics to the trial.

A huge thanks to our authors, editors and reviewers who have engaged with us in this journey, we couldn’t have done it without you! We hope that this initiative, alongside others that promote data and protocol sharing, will help us develop our articles to live by the promise of more open and reproducible science.

References

1. Pastrana, E., Kousta, S. & Swaminathan, S. Three Approaches to Support Reproducible Research. Science Editor

AUTHORS: This guest blog comes from Erika Pastrana, Editorial Director for the Nature Research Journals and Sowmya Swaminathan, Head of Editorial Policy and Research Integrity at Nature Research.

Peer Review Week 2019: Improving peer review quality through transparent, reproducible research

Posted on by

This guest blog comes from Sowmya Swaminathan, Head of Editorial Policy and Research Integrity for Nature Research

By the time a research study reaches the peer review process, many crucial decisions that affect the rigor of the study design, methodology, data collection, analysis and reporting have already been made. Nevertheless, by developing and implementing editorial policies and by providing a publishing infrastructure that supports publication of transparent reproducible research, editors, journals and publishers can help improve the published paper, adding value and quality to the peer review and publication process.

Broadly speaking, four pillars – policy, publishing infrastructure, advocacy and awareness, and collective action – have driven editorial and publishing innovation and furthered our mission to work in partnership with the research community to advance quality and integrity.  In this blog post I provide an overview and examples of the many initiatives undertaken at Nature Research to support publication of reproducible research.

Policy

Transparency is at the heart of our policies designed to improve the reproducibility of published research. We ask authors to report information about their experimental design, as well as to clearly identify and make their datasets, code and materials available, also making it easier for reviewers to access the information they need to assess the study appropriately. We strongly support open research practices such as sharing the underlying building blocks of the research article – data, code and protocols – through repositories.

We have found that policies centred on transparency have had an impact.  For example, independent studies have found that the Nature Research Life Science Reporting Summary, an instrument to support transparent reporting in life science articles, which we introduced in 2013, has improved reporting of statistics and other aspects of experimental design and analysis [1,2].

We recognize that what works for reporting in life sciences is often not applicable to many of the other disciplines. While we advocate for a minimum threshold for transparency across core aspects of data, code, and materials, we have also worked with experts to tailor approaches that are designed to meet field-specific needs, for example in areas of photovoltaics and photonics research.

Data availability is another area where implementing a policy focused on transparency has had clear benefits. Since 2016, when we introduced a mandatory data availability statement on all research articles published in Nature-branded journals, we have seen a rise in data sharing through public repositories across our journals, especially in the life sciences, and increased appreciation of the value of data sharing to underscore the integrity and credibility of published work in many disciplines.

Publishing Infrastructure

Designing an innovative peer review and publishing infrastructure that supports all aspects of publishing reproducible research is central to our overall vision for an open and transparent ecosystem. A robust technology infrastructure is also essential to drive large-scale adoption of best practice approaches by authors, reviewers and editors. Over the years, we have introduced a number of publishing innovations that have furthered our commitment to reproducibility. These include avenues for publishing data and protocols such as Scientific Data and Protocol Exchange, and new article formats like Data Descriptors and Registered Reports that focus on data and methodological rigour respectively, rather than the specific results.

More recently, three Nature Research journals have tested executable platforms for peer review and publication of code. Although the policy and practice of peer reviewing code has been in place at these journals for many years, powering the process through an executable platform sets the stage for a more seamless and scalable experience for authors, reviewers and editors.

Advocacy and awareness

Advocacy and awareness-raising in the broader research and publishing community are other important areas of engagement for us in advancing our commitment to integrity in research. In the pages of Nature and the Nature-branded journals, we have often highlighted and debated the many different, complex issues, challenges and solutions on the path to transparent, reproducible research including discipline-specific needs and barriers to reproducible research (for example, see recent discussions about reproducibility in nano-medicine and data and code sharing in physics).

Collective action

Shifting entrenched patterns of how research is conducted and published requires stakeholders across the research and publishing community to work collectively in the push for better practice. Nature Research journals are proud to have participated in and supported many such efforts to accelerate data sharing, advance best practice toward open and transparent research and align on minimum reporting standards.

We believe that our editors and journals have an important role to play in tackling the many issues that affect the quality and integrity of published research. Indeed, we feel privileged to be able to engage with a global and multidisciplinary research community and are committed to furthering the cause of transparent, reliable research with all the tools at our disposal.

Join the discussions during Peer Review Week: #QualityinPeerReview, #PeerRevWk2019 #PeerReviewWeek

References:

  1. The NPQIP Collaborative group, Did a change in Nature journals’ editorial policy for life sciences research improve reporting? BMJ Open Science 2019;3:e000035. doi: 10.1136/bmjos-2017-000035
  2. Han S, et al. (2017) A checklist is associated with increased quality of reporting preclinical biomedical research: A systematic review. PLoS ONE 12(9): e0183591. https://doi.org/10.1371/journal.pone.0183591