This post is by Myles Allen of the University of Oxford:

As a vocal supporter of the traditional system of scientists communicating through peer-reviewed channels ( Nat. Geosci. 1, 209; 2008 and associated debate at Peer-to-Peer), I was hesitant about writing a critical Commentary on the Feature ‘A safe operating space for humanity’ by Johan Rockström et al in the 24 September issue of Nature (Nature 461, 472-475; 2009) in a non-peer-reviewed forum. The Nature and Nature Reports Climate Change editors had clearly thought through this argument: the Feature was not itself peer-reviewed, so no golden rules would be broken in publishing a series of commentaries alongside it in Nature Reports Climate Change.

The problem is that packing the point into a few hundred words, and slipping into the usual bloggers’ trap of feeling you have to shout loudly on the internet or no-one will listen, means that the end result reads very black-and-white. Corresponding with Johan Rockström over the past week, it emerges we agree on far more than the tone of my Commentary, ‘Tangible targets are critical’ (Nat. Rep. Climate Change doi:10.1038/climate.2009.95), probably implies. I understand that Rockström et al. had not originally intended to make the link between a six-degree climate sensitivity, the two degrees target and 350 p.p.m. a focal point (as I read it) of their Feature. Likewise, they observe, reasonably enough, that limiting cumulative carbon dioxide emissions to one trillion tonnes of carbon is just another way of framing the climate boundary, with (as I acknowledge) remarkably similar implications to 350 p.p.m.

If this had been done the old-fashioned way, Rockström et al. would almost certainly have had to qualify their reliance on a six-degree climate sensitivity in the course of the usual to-and-fro with referees. And I in turn would have toned down a lot of my objections. The end result would undoubtedly have been blander on both sides, but would that really have been much of a loss? Following what was, for me, an experiment, I still feel it is very much an open question whether scientific communication in general benefits from direct publication rather than allowing rough edges to be smoothed off through traditional peer-review.

Myles Allen is at the University of Oxford, Parks Road, Oxford, OX1 3PU, UK.

–—————————————

Context:

Johan Rockström and his co-authors argue in a Nature Feature that to avoid catastrophic environmental change, humanity must stay within defined ‘planetary boundaries’ for a range of essential Earth-system processes. If one boundary is transgressed, then safe levels for other processes could also be under serious risk, they caution. Seven expert commentaries respond to this proposal in Nature Reports Climate Change, one of which is by Myles Allen. All these articles can be accessed from this index page.

There is an associated Nature podcast in which Johan Rockström is interviewed and editor Ehsan Masood provides further analysis.

Further reading:

A longer paper upon which the Nature Feature is based, ‘Tipping towards the unknown’, is available at the Stockholm Resilience Centre.

Nature news special: The road to Copenhagen.

Manuscript peer reviewing is at the heart of the scientific system, but it seems that these duties are often not properly (if at all) recognized by universities, funding agencies or even the rest of the scientific community. This is the main message of the September Editorial in Nature Structural & Molecular Biology, ‘The Unsung Reviewer’ (16, 899; 2009) The Editorial notes:

“Scientists wear many hats these days. They prepare and teach classes and sit on various committees. Then there are the multiple activities directly related to their research, including grant writing, mentoring students and postdocs, attending conferences, writing papers and reviewing manuscripts. All these duties can take a considerable amount of time and effort and most are recognized as worthy contributions by funding agencies, universities and research institutions when evaluating a scientist’s performance. On the other hand, peer reviewing papers seems to be the Rodney Dangerfield (”I get no respect!“) of a scientist’s duties.”

A new report by the Science for Policy project of the US Bipartisan Policy Center notes that peer-review is essential for the scientific system. Journals such as NSMB (and the other Nature journals) would not agree from their own perspective with one of the report’s conclusions, that “peer-review is no longer assumed to be a professional obligation”. As the Editorial points out, our journals have very broad reviewer pools (running into many tens of thousands for the heavily oversubscribed weekly title Nature, for example), and the editors know and very much appreciate the work that these reviewers put into improving submitted manuscripts.

The SPP report also identifies a need to increase the number of scientists who participate in peer review for federal agencies, making suggestions such as listing such service on grant applications or even making it a requirement for funding. In addition, it urges journals to run a quality system, for example by providing the peer-reviewers with feedback (which the Nature journals do, and which is appreciated by our reviewers – for one reason, because a reviewer can see his or her report in the context of those of the other reviwers of the manuscript). The NSMB Editorial concludes:

“Peer review is often compared to jury duty, a chore that one has to do once in a while as a service to the community. But reviewing manuscripts can be enjoyable and useful at any career stage. True, it takes time, but as one reviewer aptly put it: “I still learn do’s and don’ts from reviewing papers, as well as just getting useful information, so it is time spent pretty well.” It is also true that one does not get much open recognition from it, and there are limitations on what the journals can do without compromising the reviewers’ anonymity. We have in the past publicly acknowledged our reviewers at the end of the year, and will do so again this year. The SPP report stresses that it is important to stress that “peer reviewing manuscripts should be an expected and appreciated aspect of a scientist’s career.” Of course, we do have a vested interest in this, but we firmly believe that it’s important to cultivate a vibrant ‘reviewer culture’."

Nature journals’ peer-review policies and guidelines.

More on peer review from the Nature journals.

Brian Derby writes a post with the title Refereeing Chores at his Nature Network blog. Brian has refereed hundreds of papers in his scientific career, so he’s as experienced as it gets at the process. Even so, he writes about how it took half a day to reach his decision – to recommend rejection of the submitted paper – and had yet to write his report in suitably critical yet constructive tones for the author and journal.

The process required a first read to form an initial impression, a second read, to use Brian’s words, “required that I check up on some of the references and make sure that the article is consistent, which it was but I did not agree on the applicability of the model proposed. More reading of the bibliography. I discover at least one self-reference that appears to be there for no reason other than to (perhaps) improve citation score (I am now grumpy). I discover another reference that is on a completely different subject that must be in by mistake…. A discreet phone call to a colleague allows me to discuss the mechanism and we agree that it is flawed. Now follows the difficult bit, I have to write a carefully worded letter justifying my concerns to the editor.”

Or to put it another way, here is how the Nature journals recommend a review is written:

“The primary purpose of the review is to provide the editors with the information needed to reach a decision. The review should also instruct the authors on how they can strengthen their paper to the point where it may be acceptable. As far as possible, a negative review should explain to the authors the weaknesses of their manuscript, so that rejected authors can understand the basis for the decision and see in broad terms what needs to be done to improve the manuscript for publication elsewhere. This is secondary to the other functions, however, and referees should not feel obliged to provide detailed, constructive advice to authors of papers that do not meet the criteria for the journal (as outlined in the letter from the editor when asking for the review). If the reviewer believes that a manuscript would not be suitable for publication, his/her report to the author should be as brief as is consistent with enabling the author to understand the reason for the decision.

Confidential comments to the editor are welcome, but it is helpful if the main points are stated in the comments for transmission to the authors. The ideal review should answer the following questions.” (More than a dozen follow, and this list is followed by a second list of “further questions to consider”.)

The Journal of Cell Biology has launched an application that is intended to transform the way researchers can store, share and access the growing number of microscopy images. (See E. Hill, J. Cell Biol. 183, 969–970; 2008). The service is described in a Nature online news story on 19 December 2008.

Cell and molecular biology laboratories routinely produce image files in a wide variety of formats, and researchers often keep images on their own hard drives. The new JCB DataViewer, supported by the University of Rockefeller Press, publisher of the journal, is a database into which all authors of papers published by the Journal of Cell Biology have to submit their raw images — which can amount to a terabyte or more of data. The software then allows others to retrieve and manipulate the data through a web browser. As well as the ability of the system to store these huge datasets, the hope is that the JCB DataViewer will help peer reviewers and others in the community judge the quality of the work.

From the Nature news story: “It’s transparency and integrity,” says Jason Swedlow, a microscopy specialist at the University of Dundee, UK, who led much of the technical development. “It is going to allow you to see that data and judge it for yourself.” The journal’s editors hope that "eventually a version of this database could become a central repository for image data from the worldwide research community — something like a microscopy version of Genbank, a database in which vast amounts of genetic-sequence data are shared and stored. In future, research funding agencies and journals might require that authors submit their microscopy data to such a repository, just as most require submissions of DNA sequences to Genbank or other similar banks. “There have been murmurings about wanting data to be kept and maintained somewhere but no one has mandated it yet,” Hill says."

A news story in Nature Medicine (14, 1133; 2008) discusses an investigation into the publication status of the clinical-trials literature (PLoS Med., doi: 10.1371/journal.pmed.0050191; 2008), which concludes that positive results of clinical trials for drugs or devices have a higher chance of getting published than negative trials. The study further concludes that when the trial sponsors publish the results from ‘pivotal’ trials only 76% of the time.

Studies reporting a statistically significant difference were more than three times as likely to be published. This ‘positive publication bias’ is a serious problem, because it can make a drug or device appear in the literature to be more effective than it is. Ida Sim, a co-author of the study, told Nature Medicine: “We have this idea of practicing evidence-based medicine, which is predicated on having a full and complete evidence base. But when the evidence base is skewed, we can’t really do this.”

According to the Nature Medicine article, a paper in Science (319, 1340–1342; 2008) indicates that “the FDA Amendments Act of 2007 has improved transparency, because the law mandates that sponsors or primary investigators of clinical trials for approved drugs post a summary of their results in a national open-access database. The lead author of the report, Deborah Zarin, oversees the ”https://clinicaltrials.gov/">ClinicalTrials.gov registry at the National Library of Medicine of the US National Institutes of Health and is in charge of ensuring the results are posted in compliance with what the new law. According to Zarin, “for the trials that are covered by this law, the results database should have a big impact on disseminating medical knowledge, because the results have to be publicly available.”

But not every type of clinical trial is covered by the legislation, nor does it directly affect medical journals. Although Sim applauds the FDA Amendments Act of 2007, she adds that it “doesn’t address the issue of not publishing trials in medical journals. They remain one of the most influential and biased sources of information.” "