Nature Medicine peers into review

Posted on 22 Mar 2010 by Maxine Clarke

In its March Editorial, Peering into review, Nature Medicine addresses how the peer review process can be frustrating to researchers eager to get their work published. Changes to the process might be warranted, says the journal —but only if they are based in fact, not conjecture.

The Editorial discusses a recent “open letter” written by a group of stem-cell researchers about what they see as obstructive and unreasonable reviews delaying publication of their research. From the Editorial:

Publication of referees’ comments in full may affect the quality of the reviews, leading to more cautious and restrained comments. It is difficult to ascertain how much the quality of reviews would be compromised by adopting these measures; however, previous attempts with open peer review suggest that referees are less likely to provide a direct and detailed evaluation of the report. Authors may also be reluctant to adopt this strategy, as publication of earlier reviews may expose flaws that were addressed in later submissions (or never addressed satisfactorily) but nonetheless color interpretation of the findings. Finally, most articles undergo numerous rounds of review. As such, it is difficult to evaluate the importance of comments made in the first round of review without reference to that earlier version of the manuscript. Simply publishing the reviewers’ comments as a whole removes them from the context in which the decisions were made and may place undue emphasis on specific points of concern.

We do not want our readership to feel that the editors are indifferent to the concerns voiced by the stem cell researchers. The peer review process might be improved, but it remains unclear whether publishing reviewers’ comments would provide a measurable advantage that outweighs the drawbacks of adopting this approach. When properly managed, confidential peer review allows for a fair and unbiased assessment and ensures that we publish papers of the highest quality.

Comments from readers are invited at Spoonful of Medicine, the journal’s blog.

Nature Chemistry on improving peer review

Posted on 26 Oct 2009 by Maxine Clarke

Perceived lapses in the peer-review process often receive a lot of attention, but the majority of researchers declare themselves satisfied with the system even though they would like to improve it. If it is imperfect or broken, how do we fix it? This question is addressed in the November Editorial of Nature Chemistry ( 1, 585; 2009), in light of some blog commentaries which identified prior publications that had not been referenced in a journal paper.

Open peer-review experiments have generally not been very successful because reviewers are less likely to make forthright comments in an open forum. Double-blind peer review is another option, but one must consider the role of the editor who oversees the process, as well as the difficulties of effectively hiding the identity of authors in smaller fields from other experts — especially when many authors regularly cite and discuss their previous work. The Editorial concludes:

“The Royal Society of Chemistry’s Dalton and Faraday discussion meetings provide a unique mix of traditional peer review coupled with both comment (by peers) and responses from the authors, but require members of a particular research community to assemble at a conference. It is in some ways similar to the grant proposal review process at, for example, the US ”https://grants.nih.gov/grants/peer_review_process.htm">National Institutes of Health. However, such a process is clearly not a viable option for every one of the vast number of papers submitted for publication. The journal Atmospheric Chemistry and Physics uses a system in which, after initial assessment by an associate editor, manuscripts are posted online for comment. After referee reports are received, these are also posted online with the manuscript along with author rebuttals. If eventually accepted, a paper is formally published in the journal, whereas those that are not remain available (and citable) as online ‘discussions’. This differs from the preprint servers Nature Precedings and arXiv because there is an initial assessment of the suitability of the work (based on more than just scope).

Perhaps a hybrid system could be the solution. Traditional peer review, and a decision to publish, could be followed by a fixed period in which any interested party could post questions or comments and the authors are given the opportunity to respond — all moderated by an editor — before a final version of the article (including comments and responses) is preserved for the record. This would again require a large change in the habits of the community — authors, reviewers and publishers — and previous experiments with commenting on published papers have been far from conclusive."

Sense about Science peer-review survey 2009.

Nature journals’ peer review policy and Editorials on the subject.

Hoax paper accepted for publication

Posted on 18 Jun 2009 by Maxine Clarke

A story published in Nature News online on 15 June, describes how the editor-in-chief of a journal is to resign after claiming that the publisher, Bentham Science Publishing, accepted a hoax article for publication without his knowledge.

From the Nature News story:

The fake, computer-generated manuscript was submitted to The Open Information Science Journal by Philip Davis, a graduate student in communication sciences at Cornell University in Ithaca, New York, and Kent Anderson, executive director of international business and product development at The New England Journal of Medicine. They produced the paper using software that generates grammatically correct but nonsensical text, and submitted the manuscript under pseudonyms in late January. After receiving several unsolicited invitations by e-mail to submit papers to open-access journals published by Bentham under the author-pays-for-publication model, Davis wanted to test if the publisher would “accept a completely nonsensical manuscript if the authors were willing to pay”. The manuscript was accepted with a request from the publisher for Davis to pay US$800 to its subscriptions department, based in the United Arab Emirates, before the article was published. Davis then retracted the article.

Bambang Parmanto, an information scientist at the University of Pittsburgh, Pennsylvania, and editor-in-chief of The Open Information Science Journal, told Nature that he had not seen the manuscript or any peer review comments before it was accepted. Nor was he informed that the manuscript had been accepted for publication. “I think this is a breach of policy,” he says, adding: “I will definitely resign. Normally I see everything that comes through. I don’t know why I did not see this. I at least need to see the reviewer’s comments.” Parmanto says that Bentham Science Publishing told him that the manuscript had been reviewed by one member of the journal’s editorial board. “The peer review didn’t work,” says Parmanto, who now fears that the journal’s publishing system could be open to abuse. “The publisher could take advantage of the fees, and that is why I want to leave,” he says.

Mahmood Alam, director of publications at Bentham Science Publishing, told Nature in an e-mail statement that “submission of fake manuscripts is a totally unethical activity and must be condemned.” He defended Bentham’s peer review process, saying, “a rigorous peer review process takes place for all articles that are submitted to us for publication. Our standard policy is that at least two positive comments are required from the referees before an article is accepted for publication.” In this particular case, “the paper was reviewed by more than one person”.

Peter Suber, a philosopher at Earlham College in Richmond, Indiana, and a proponent of open-access publishing, is worried that the case could turn people against the author-pays open-access model. “There are many legitimate and rigorous open-access journals that use this same business model,” he says.

Further details are provided in two articles at The Scholarly Kitchen blog: an account of the experiment by Philip Davis, Nonsense for dollars; and an editorial, The tip of an Iceberg, by Kent Anderson.

Janet Young, in a comment to the Nature News story, writes: “I’ve had six requests to review papers from a Bentham Science journal over the last year. The first was for a paper in my field, but I refused as I was very busy at the time. The other five have been for papers in fields I know nothing about – I would have been an utterly inappropriate reviewer had I accepted the requests (I didn’t). Each request had the full paper attached to the email, rather than just an abstract. That seems like a very unusual review process to me.”

See Nature’s news website for the full version of the article, and to add your own comments. You are also welcome to comment here.

Websites encourage direct public funding for research

Posted on 21 May 2009 by Maxine Clarke

The ‘SciFlies’ project, according to a Nature news story (Nature 459, 305; 2009), will profile scientists from a range of disciplines and the new ideas they want to pursue, or ways in which they would like to expand their current research programme. Website visitors will be able to donate any amount to support the projects they find most interesting or worthwhile.

The website itself states: “We look forward to receiving your application for funding of initial proof-of-concept STEM research projects in the range of $5,000 to $12,000. To participate in this unique online grassroots-funded opportunity, please complete the questionnaire about your project, including details of its possible outcome/impact and profiles of the researchers or research team. SciFlies.org will then depict the project for online users to view and decide if they want to make a charitable contribution in support of the project-funding goal. Once each project’s funding goal is reached, researchers will be notified and the process of funding will be completed based on a mutually acceptable agreement of terms between researcher and SciFlies.org….Projects are competing for support from the general public, so researchers are encouraged to describe their work in appealing and accessible terms, such that users can easily understand the concepts and potential outcomes. Avoid “science-speak,” acronyms or abstract language. A proposal that conveys the researcher’s excitement about a project and its potential, as well as providing insight into his/her personal story, can significantly attract donors.”

At this stage, there is nothing on the website about the peer-review or other assessment process, but there are already some projects listed. A full launch is promised for mid-July.

According to the Nature News story, “David Fries, a marine engineer at the University of South Florida in St Petersburg, conceived of and heads the SciFlies effort. His main inspiration was long-standing frustration with a research funding structure that, with few exceptions, offers scientists no intermediate steps on the way to requesting full grant funding.” Daniel Gaddy, in a comment to the News story, mentions a similar enterprise called FundScience, which also seeks direct public funding of research projects. FundScience provides a description of its activities here, but although a press release is also provided, there is no information about whether a form of independent peer-review will be used to aid potential donors.

Nature Neuroscience experience with peer-review consortium

Posted on 31 Mar 2009 by Maxine Clarke

In 2008, the journal Nature Neuroscience joined a newly created community consortium aimed at making peer review more efficient by allowing reviews to be transferred between consortium journals. In its current (April) issue, the editors look back at their experience with the Neuroscience Peer Review Consortium over the past year (Nature Neuroscience 12, 363; 2009).

Journals in the Neuroscience Peer Review Consortium (NPRC) offer authors whose papers are no longer under consideration at a journal an opportunity to transfer reviews of their manuscipts when submitting their paper to another consortium journal. After a year, Nature Neuroscience‘s experience is similar to that of other journals in the consortium, with only a handful of papers being transferred from Nature Neuroscience to another consortium journal.

Similar to the Nature journals’ transfer system, the NPRC system is voluntary for authors and referees. Editors at one journal only know that a paper was reviewed elsewhere if the author chooses to inform them. At Nature Neuroscience, the editors ask referees for permission to release their identities whenever authors ask for their papers to be transferred to another consortium journal. If a reviewer declines to participate, the reviews (comments to authors only) are transferred anonymously.

All the transfers from Nature Neuroscience to date have been to the Journal of Neuroscience, and represent less than 1% of manuscripts that are eventually rejected after review. However, for the papers that were eventually published in the Journal of Neuroscience, the authors reported that the paper had been expedited. Even in the cases where new referees were solicited, authors felt that transferring the reviews from Nature Neuroscience had saved them time and effort.

No papers have been transferred to Nature Neuroscience from other consortium journals.

The Nature Neuroscience editors ask why so few authors are using the NPRC option. They conclude: “Authors may simply not be aware of NPRC or may not know what journals participate in it. Transfer rates may pick up as more authors learn of the consortium. At Nature Neuroscience, we have noticed an increase in the number of referees that state in comments to the editors whether they wish for their identities to be released to other consortium journals or not, suggesting a growing awareness of the NPRC.

It could also be that there are not that many papers that lend themselves well to this process. Many of our authors who have had papers rejected may prefer to take their chances with new referees at another journal, rather than making substantial revisions in response to the concerns raised by our referees. Certainly, our authors appear to be more conservative when deciding to transfer their reviews, preferentially choosing to utilize the NPRC transfer option when the reviewers reject the paper on conceptual grounds and not for technical reasons.

Another factor that influences the success of the transfer is whether the referees allow the release of their identities to receiving consortium journals. Previous reviews are clearly less useful to the receiving journal if the editors do not know who the reviewers were.”

Nature Neuroscience concludes that it is premature to gauge whether the system truly could save referees, authors and editors substantial time and effort. The editors encourage authors, referees and readers to share their views, either by email or by commenting here.

Journal of Biology adds a twist to peer-review

Posted on 29 Jan 2009 by Maxine Clarke

The Journal of Biology (8, 1; 2009) has announced an experimental policy of allowing authors of submitted manuscripts to opt-out of re-revew on occasions where the peer-reviewers require revisions, including the addition of data. In these cases, the journal will not publish the referees’ reports with the manuscript, but instead will publish an accompanying commentary.

This adds a new model of peer-review to those previously described in Nature‘s peer-review debate of 2006. Journals using unusual forms of peer-review system include Biology Direct, Atmospheric Chemistry and Physics, Signaling Gateway (a database publication), Electronic Transactions on Artificial Intelligence, and the BioMed Central (BMC) journals – BMC publishes Journal of Biology. Further discussion on systems of peer-review since 2006 is archived in a series of posts on this blog.

It will be interesting to see how the Journal of Biology experiment is received by readers and peer-reviewers (no doubt that it will be popular with authors). As well as the question of the accuracy of pubished papers whose authors have not addressed technical criticisms, there is also the question of motivation of peer-reviewers to write detailed reports, if they know their advice can, if the author wishes, go unheeded. Particularly strange to me is the decision by the journal not to publish the referees’ reports with the unrevised manuscript, but instead to publish an independent commentary. Will this commentary always accompany publication of incomplete manuscripts, or might it be delayed? Will it be written by one of the peer-reviewers, or if not, by someone who has access to the reports?

The Nature journals’ peer-review policy and procedures are described here. This web page contains an archive of free-to-access editorials in many Nature journals that discuss aspects of the peer-review system and process.

How to deal with technical criticisms of published work

Posted on 16 Jan 2009 by Maxine Clarke

“Scientific publishing depends on expert peer reviewers. Instead of perpetually arguing about the reliability and fairness of peer review, authors, editors and referees should seek to optimize this time-tested system.” So opens the January editorial of Nature Neuroscience (12, 1; 2009).

The editorial discusses the media reporting of a ferocious argument about the merits of a paper published in Cell , and a subsequent blog debate hosted by The Scientist. The controversy between scientists in this discipline concerning this paper “has again ignited a debate on the flaws of editor-managed anonymous peer review”, write the Nature Neuroscience editors. “We maintain, however, that despite occasional unfortunate lapses, anonymous peer review remains the best quality-control process that we have.” The editorial goes on to discuss how journals can best optimize the process.

In Nature News this week (457, 245; 15 January 2009) another technical dispute is discussed, this time concerning a widely circulating preprint attacking much of the published research in social neuroscience involving functional magnetic resonance imaging (fMRI). This affair is exacerbated by the fact that the circulating preprint is not due to be published, with responses from the authors of some of the criticized studies, for another nine months, according to the Nature News story. The criticisms have already been covered in publications such as Newsweek, as well as the inevitable blogosphere outpourings – and at least some of the criticized authors say the first they heard of the preprint was when they were contacted by journalists.

How scientific reports should be peer-reviewed and, after publication, scrutinized are matters that are decided upon by the journals, their editors, and their publishers and/or societies – in the case of articles submitted to and published in the Nature journals, the peer-review process is described here, and the post-publication corrections process here. Good journals have processes for investigating technical criticisms and complaints about the papers they publish. Nature‘s, for example, is here. Often, a resolution is not clear-cut at the outset, when the complaint is first received by the journal, however clear it may be in the mind of the complainer. A proper outcome depends on independent peer-reviewers, as well as editors, examining the complaint together with a measured response from the study’s authors. Playing out such investigations in the kangaroo courts of the popular press, or in unfettered comments on the Internet between people who have been described as “recreationally outraged”, not only obscures logical, technically informed investigation, but unnecessarily exacerbates emotions and arguments so that, in the end, all that is remembered is the heat – not any light.

EMBO journal introduces transparent peer-review

Posted on 05 Jan 2009 by Maxine Clarke

Via press release, The EMBO Journal will be publishing online author and referee comments from this year (2009). “The EMBO Journal has been our flagship publication for 27 years, sharing knowledge broadly within the molecular life sciences community,” said Hermann Bujard, director of EMBO (European Molecular Biology Organisation). “We are excited by the editorial changes that will make publication of research findings more transparent, complete and visible.”

By making the evaluation of manuscripts visible to everyone, The EMBO Journal aims to encourage constructive referee and author argumentation. Younger scientists will gain valuable insight into how to publish their research findings as well as how to deal with criticism.

“The EMBO Journal has an efficient and reasonable editorial process,” writes Executive Editor Pernille Rørth in an advanced online publication of the editorial in the first issue for 2009 ”https://www.nature.com/emboj/journal/vaop/ncurrent/full/emboj2008250a.html">(EMBO J. 28 , published online 4 December 2008). “A transparent editorial process will help demystify decisions.”

Beginning with manuscripts submitted in 2009, a supplementary process file will be included with the online publication of papers. This file will show all dates relevant to manuscript processing and communications between the author, editors, referees and comments to the decision letter. Readers will learn about why referees find the paper interesting, any gaps they may have identified in the initial research findings, and how the gaps were resolved in revision. Referee identities will be anonymous and confidential comments between the referee and editors will remain so. Authors will have the option to decline publication of the editorial process when they submit manuscripts, but are encouraged to participate.

Instructions for authors are available here.

NIH plans to streamline processing of grant applications

Posted on 14 Nov 2008 by Maxine Clarke

The US National Institutes of Health (NIH) will introduce new guidelines in January 2009 allowing biomedical researchers to amend and resubmit a failed funding application only once, as part of an overhaul of the peer-review system for evaluating grant proosals (see Nature 455, 841; 2008). Applicants whose grants are unfunded after the second submission may reapply only after designing a new proposal. NIH has previously suggested not allowing resubmissions, but decided against this step after an outcry from researchers (see Nature 453, 835; 2008).

NIH estimates that the move will reduce the number of applications by up to 5,000 — welcome news as it struggles to evaluate about 55,000 applications this year. In 2007, only about 30% of awards were granted to first-time submissions.

According to a comment to the Nature News story by Jeremy Green, in the current UK funding system, the research councils and the Wellcome Trust operate a no re-submission policy, although depending on the subject area an applicant might be able to submit a proposal rejected by one funder to the others. However, applicants to the research councils do get an opportunity before the grant review panel sits to see and respond to the reviewers’ comments.

NIH responds to critics

Posted on 12 Jun 2008 by Maxine Clarke

A News story in the 12 June issue of Nature (453, 823; 2008) by Meredith Wadman:

Responding to hundreds of critical comments, the US National Institutes of Health (NIH) has reversed several controversial proposals made in February as part of a year-long effort to overhaul the agency’s peer-review system (see Nature 451, 1035; 2008).

As part of an initiative called Enhancing Peer Review, announced in a finalized form on 6 June, the agency will spend at least $200 million annually over the next five years to foster groundbreaking, investigator-initiated research. Of that, at least $250 million will go to a new beast: a Transformative R01 Award, a reach-for-the-skies version of the NIH’s basic grant. The remaining $750 million will go to existing awards that reward risk and innovation: the Eureka, New Innovator and Pioneer awards.

The changes “are concrete solutions that will maximize flexibility, remove any unnecessary burden, stimulate new innovation and promote transformative research”, says NIH director Elias Zerhouni.

They include rewards for long-serving reviewers; a streamlined, 12-page R01 grant application, down from 25; and a seven-point, integer scoring scale for grant applications, which will be assessed across five criteria: impact, investigators, innovation, feasibility and environment. Current applications are graded on a 41 point scale, from 1.0 to 5.0, raising complaints that they claim a degree of accuracy that can’t be scientifically defended.

Among the controversial proposals shelved by the agency was a recommendation that all applications, even those on a second or third submission, would be treated as new, without reviewer access to prior reviews.

Gone, too, is the category “not recommended for resubmission”, which had been suggested for dismal applications. Scientists felt that branding projects with “a clear, checkbox-driven stigma is bad, that it could have unintended consequences”, Jeremy Berg, director of the National Institute of General Medical Sciences, told the advisory committee.

Berg and Lawrence Tabak, director of the NIH’s dental institute, head the group that developed the recommendations and are charged with implementing them over the next 18 months.

The agency also jettisoned a “minimum effort requirement” that would have required principal investigators to commit at least 20% of their time to any single NIH grant — an item of particular concern for ‘grandee grantees’ (see Nature 452, 258–259; 2008). Instead, grantees will need to indicate if they will have more than $1 million in cumulative NIH funding.

nature.com blogs

Category Archives: Systems