In Nature‘s February technology feature on ‘deep learning‘, a kind of artificial intelligence whose usage is spiking in life science research, author Sarah Webb points readers to a ‘comprehensive, crowd-sourced’ review of the field.
Available as a preprint on bioRxiv (ETA: and now online in the Journal of the Royal Society Interface), the review is indeed comprehensive: the PDF runs to 123 pages and 552 references, and has been downloaded nearly 27,500 times since May 2017. But it was an intriguing footnote on the article’s title page that really piqued my interest: “Author order was determined with a randomized algorithm”.
That’s the kind of footnote one simply has to follow up on, so I contacted the review’s two senior authors, Casey Greene of the University of Pennsylvania Perelman School of Medicine, and Anthony Gitter of the University of Wisconsin-Madison.
As it turns out, the review (“Opportunities and obstacles for deep learning in biology and medicine”) — which the authors call the ‘Deep Review’ — was a crowdsourced effort induced by a single tweet. In August 2016, Greene, a computational biologist who was struggling with the breadth of the deep learning field, took to social media to invite his colleagues to join him in writing a review:
“Enjoy the #Genomics and #deeplearning literature and want to share thoughts? We’re writing a review via #github!” — 9:55 AM – 5 Aug 2016
More than 40 researchers answered the call, contributing everything from summaries of key papers, to entire sections, to grammar checks — an example of what can happen as open-source software sensibilities meet open-access science. Thirty-six of those individuals contributed enough to merit authorship on the final article.
“There were some people who had specific expertise that we thought was valuable that we did actually kind of hunt down and say, like, ‘please do this, please do this, please do this’,” Greene says. “But a lot of the contributions arose a little bit more organically.”
In the traditional model of manuscript preparation, one or a few individuals write a first draft in Microsoft Word, Google Docs, or similar software and send it around for comment. Online collaborative authoring tools such as Authorea and Overleaf, which I covered in a 2014 Toolbox, can simplify the process. But for their review, Greene and Gitter went old-school, embracing the tools of software development to produce their manuscript as a plain-text file on the code-sharing service, GitHub. Anybody could contribute text or corrections in the form of a ‘pull request’, a GitHub mechanism normally used to suggest changes in software code. But those changes weren’t simply accepted without vetting.
Upon receipt of a pull request, a custom-built software tool called ‘Manubot‘ sprung into action. Authored by Greene lab postdoc Daniel Himmelstein, Manubot tests the modified document for formatting errors and retrieves citations, kicking it back to its author in the event of an error. The change then gets passed to one of the project leaders, who can approve and ‘merge’ it into the final document. Finally, Manubot is executed again to reformat the final document, rebuild the author list, and upload the revision to GitHub. The software even logs the new version in the Bitcoin blockchain, establishing a definitive record of its history.
“The [GitHub] workflow parallelizes well,” says Himmelstein. “And it’s a workflow that big open-source projects like Linux and Python use, where they have thousands of contributions potentially ongoing at a single time.”
Indeed, Manubot turned out to be so useful that Himmelstein spun it out into a standalone project, which anyone can download and use. Greene’s team has used it to create a number of public documents, including an article detailing the creation of the Deep Review (which Greene calls the ‘Meta Review’), a grant proposal to fund Manubot development, and an analysis of Sci-Hub publication coverage.
But back to the Deep Review author list. As Greene explains, GitHub is based on Git, a version-control software tool that makes it easy to establish which individual contributed what content. But with so many moving parts it was difficult to fairly determine which authors should receive precedence. So, the team opted for a randomized approach.
In order to receive authorship, participants had to submit substantive pull requests on GitHub, contribute to the overall design of the manuscript, and approve the final document. Authors who made that cut were partitioned into four categories based on their level of participation — contributing an entire subsection, for instance, or providing smaller bits of text — and the order within each section determined randomly using a Python script in a Jupyter notebook (available here; see also an ancillary notebook that computes each individual’s contributions).
Even the corresponding authors were randomized, with Greene emerging as the (reluctant) final author. Despite his best efforts to give Gitter the place of honor, Greene says, Gitter demurred: “No, the notebook has spoken.”
Not every article can be written this way, of course — the approach works best for crowdsourced projects that are open from the get-go, Greene says, as opposed to those that are written privately and then published to a preprint service. And because the process is dependent on Git, which has a notoriously steep learning curve, it may not be for everyone.
But for those articles that can be Manubotted, the process can be rewarding. Indeed, the project’s open-source approach allowed the manuscript to take on a life of its own, says Gitter. “We’re all just amazed what the final product ended up being.”
Jeffrey Perkel is Technology Editor, Nature
Updated 13 April 2018: Greene’s review was published on 4 April at the Journal of the Royal Society Interface.