Peer-to-peer

Perceived lapses in the peer-review process often receive a lot of attention, but the majority of researchers declare themselves satisfied with the system even though they would like to improve it. If it is imperfect or broken, how do we fix it? This question is addressed in the November Editorial of Nature Chemistry ( 1, 585; 2009), in light of some blog commentaries which identified prior publications that had not been referenced in a journal paper.
Open peer-review experiments have generally not been very successful because reviewers are less likely to make forthright comments in an open forum. Double-blind peer review is another option, but one must consider the role of the editor who oversees the process, as well as the difficulties of effectively hiding the identity of authors in smaller fields from other experts — especially when many authors regularly cite and discuss their previous work. The Editorial concludes:
"The Royal Society of Chemistry's Dalton and Faraday discussion meetings provide a unique mix of traditional peer review coupled with both comment (by peers) and responses from the authors, but require members of a particular research community to assemble at a conference. It is in some ways similar to the grant proposal review process at, for example, the US National Institutes of Health. However, such a process is clearly not a viable option for every one of the vast number of papers submitted for publication. The journal Atmospheric Chemistry and Physics uses a system in which, after initial assessment by an associate editor, manuscripts are posted online for comment. After referee reports are received, these are also posted online with the manuscript along with author rebuttals. If eventually accepted, a paper is formally published in the journal, whereas those that are not remain available (and citable) as online 'discussions'. This differs from the preprint servers Nature Precedings and arXiv because there is an initial assessment of the suitability of the work (based on more than just scope).
Perhaps a hybrid system could be the solution. Traditional peer review, and a decision to publish, could be followed by a fixed period in which any interested party could post questions or comments and the authors are given the opportunity to respond — all moderated by an editor — before a final version of the article (including comments and responses) is preserved for the record. This would again require a large change in the habits of the community — authors, reviewers and publishers — and previous experiments with commenting on published papers have been far from conclusive."
Sense about Science peer-review survey 2009.
Nature journals' peer review policy and Editorials on the subject.

Posted by Maxine Clarke on October 26, 2009
Categories: Systems | Permalink | Comments (0)

A story published in Nature News online on 15 June, describes how the editor-in-chief of a journal is to resign after claiming that the publisher, Bentham Science Publishing, accepted a hoax article for publication without his knowledge.
From the Nature News story:
The fake, computer-generated manuscript was submitted to The Open Information Science Journal by Philip Davis, a graduate student in communication sciences at Cornell University in Ithaca, New York, and Kent Anderson, executive director of international business and product development at The New England Journal of Medicine. They produced the paper using software that generates grammatically correct but nonsensical text, and submitted the manuscript under pseudonyms in late January. After receiving several unsolicited invitations by e-mail to submit papers to open-access journals published by Bentham under the author-pays-for-publication model, Davis wanted to test if the publisher would "accept a completely nonsensical manuscript if the authors were willing to pay". The manuscript was accepted with a request from the publisher for Davis to pay US$800 to its subscriptions department, based in the United Arab Emirates, before the article was published. Davis then retracted the article.
Bambang Parmanto, an information scientist at the University of Pittsburgh, Pennsylvania, and editor-in-chief of The Open Information Science Journal, told Nature that he had not seen the manuscript or any peer review comments before it was accepted. Nor was he informed that the manuscript had been accepted for publication. "I think this is a breach of policy," he says, adding: "I will definitely resign. Normally I see everything that comes through. I don't know why I did not see this. I at least need to see the reviewer's comments." Parmanto says that Bentham Science Publishing told him that the manuscript had been reviewed by one member of the journal's editorial board. "The peer review didn't work," says Parmanto, who now fears that the journal's publishing system could be open to abuse. "The publisher could take advantage of the fees, and that is why I want to leave," he says.
Mahmood Alam, director of publications at Bentham Science Publishing, told Nature in an e-mail statement that "submission of fake manuscripts is a totally unethical activity and must be condemned." He defended Bentham's peer review process, saying, "a rigorous peer review process takes place for all articles that are submitted to us for publication. Our standard policy is that at least two positive comments are required from the referees before an article is accepted for publication." In this particular case, "the paper was reviewed by more than one person".
Peter Suber, a philosopher at Earlham College in Richmond, Indiana, and a proponent of open-access publishing, is worried that the case could turn people against the author-pays open-access model. "There are many legitimate and rigorous open-access journals that use this same business model," he says.
Further details are provided in two articles at The Scholarly Kitchen blog: an account of the experiment by Philip Davis, Nonsense for dollars; and an editorial, The tip of an Iceberg, by Kent Anderson.
Janet Young, in a comment to the Nature News story, writes: "I've had six requests to review papers from a Bentham Science journal over the last year. The first was for a paper in my field, but I refused as I was very busy at the time. The other five have been for papers in fields I know nothing about - I would have been an utterly inappropriate reviewer had I accepted the requests (I didn't). Each request had the full paper attached to the email, rather than just an abstract. That seems like a very unusual review process to me."
See Nature's news website for the full version of the article, and to add your own comments. You are also welcome to comment here.

Posted by Maxine Clarke on June 18, 2009
Categories: Ethics, Systems | Permalink | Comments (1)

The 'SciFlies' project, according to a Nature news story (Nature 459, 305; 2009), will profile scientists from a range of disciplines and the new ideas they want to pursue, or ways in which they would like to expand their current research programme. Website visitors will be able to donate any amount to support the projects they find most interesting or worthwhile.
The website itself states: "We look forward to receiving your application for funding of initial proof-of-concept STEM research projects in the range of $5,000 to $12,000. To participate in this unique online grassroots-funded opportunity, please complete the questionnaire about your project, including details of its possible outcome/impact and profiles of the researchers or research team. SciFlies.org will then depict the project for online users to view and decide if they want to make a charitable contribution in support of the project-funding goal. Once each project’s funding goal is reached, researchers will be notified and the process of funding will be completed based on a mutually acceptable agreement of terms between researcher and SciFlies.org....Projects are competing for support from the general public, so researchers are encouraged to describe their work in appealing and accessible terms, such that users can easily understand the concepts and potential outcomes. Avoid “science-speak,” acronyms or abstract language. A proposal that conveys the researcher’s excitement about a project and its potential, as well as providing insight into his/her personal story, can significantly attract donors."
At this stage, there is nothing on the website about the peer-review or other assessment process, but there are already some projects listed. A full launch is promised for mid-July.
According to the Nature News story, "David Fries, a marine engineer at the University of South Florida in St Petersburg, conceived of and heads the SciFlies effort. His main inspiration was long-standing frustration with a research funding structure that, with few exceptions, offers scientists no intermediate steps on the way to requesting full grant funding." Daniel Gaddy, in a comment to the News story, mentions a similar enterprise called FundScience, which also seeks direct public funding of research projects. FundScience provides a description of its activities here, but although a press release is also provided, there is no information about whether a form of independent peer-review will be used to aid potential donors.

Posted by Maxine Clarke on May 21, 2009
Categories: Policy and Public, Systems | Permalink | Comments (0)

In 2008, the journal Nature Neuroscience joined a newly created community consortium aimed at making peer review more efficient by allowing reviews to be transferred between consortium journals. In its current (April) issue, the editors look back at their experience with the Neuroscience Peer Review Consortium over the past year (Nature Neuroscience 12, 363; 2009).
Journals in the Neuroscience Peer Review Consortium (NPRC) offer authors whose papers are no longer under consideration at a journal an opportunity to transfer reviews of their manuscipts when submitting their paper to another consortium journal. After a year, Nature Neuroscience's experience is similar to that of other journals in the consortium, with only a handful of papers being transferred from Nature Neuroscience to another consortium journal.
Similar to the Nature journals' transfer system, the NPRC system is voluntary for authors and referees. Editors at one journal only know that a paper was reviewed elsewhere if the author chooses to inform them. At Nature Neuroscience, the editors ask referees for permission to release their identities whenever authors ask for their papers to be transferred to another consortium journal. If a reviewer declines to participate, the reviews (comments to authors only) are transferred anonymously.
All the transfers from Nature Neuroscience to date have been to the Journal of Neuroscience, and represent less than 1% of manuscripts that are eventually rejected after review. However, for the papers that were eventually published in the Journal of Neuroscience, the authors reported that the paper had been expedited. Even in the cases where new referees were solicited, authors felt that transferring the reviews from Nature Neuroscience had saved them time and effort.
No papers have been transferred to Nature Neuroscience from other consortium journals.
The Nature Neuroscience editors ask why so few authors are using the NPRC option. They conclude: "Authors may simply not be aware of NPRC or may not know what journals participate in it. Transfer rates may pick up as more authors learn of the consortium. At Nature Neuroscience, we have noticed an increase in the number of referees that state in comments to the editors whether they wish for their identities to be released to other consortium journals or not, suggesting a growing awareness of the NPRC.
It could also be that there are not that many papers that lend themselves well to this process. Many of our authors who have had papers rejected may prefer to take their chances with new referees at another journal, rather than making substantial revisions in response to the concerns raised by our referees. Certainly, our authors appear to be more conservative when deciding to transfer their reviews, preferentially choosing to utilize the NPRC transfer option when the reviewers reject the paper on conceptual grounds and not for technical reasons.
Another factor that influences the success of the transfer is whether the referees allow the release of their identities to receiving consortium journals. Previous reviews are clearly less useful to the receiving journal if the editors do not know who the reviewers were."
Nature Neuroscience concludes that it is premature to gauge whether the system truly could save referees, authors and editors substantial time and effort. The editors encourage authors, referees and readers to share their views, either by email or by commenting here.

Posted by Maxine Clarke on March 31, 2009
Categories: Quality and value, Systems, Technical solutions | Permalink | Comments (0)

The Journal of Biology (8, 1; 2009) has announced an experimental policy of allowing authors of submitted manuscripts to opt-out of re-revew on occasions where the peer-reviewers require revisions, including the addition of data. In these cases, the journal will not publish the referees' reports with the manuscript, but instead will publish an accompanying commentary.
This adds a new model of peer-review to those previously described in Nature's peer-review debate of 2006. Journals using unusual forms of peer-review system include Biology Direct, Atmospheric Chemistry and Physics, Signaling Gateway (a database publication), Electronic Transactions on Artificial Intelligence, and the BioMed Central (BMC) journals - BMC publishes Journal of Biology. Further discussion on systems of peer-review since 2006 is archived in a series of posts on this blog.
It will be interesting to see how the Journal of Biology experiment is received by readers and peer-reviewers (no doubt that it will be popular with authors). As well as the question of the accuracy of pubished papers whose authors have not addressed technical criticisms, there is also the question of motivation of peer-reviewers to write detailed reports, if they know their advice can, if the author wishes, go unheeded. Particularly strange to me is the decision by the journal not to publish the referees' reports with the unrevised manuscript, but instead to publish an independent commentary. Will this commentary always accompany publication of incomplete manuscripts, or might it be delayed? Will it be written by one of the peer-reviewers, or if not, by someone who has access to the reports?

The Nature journals' peer-review policy and procedures are described here. This web page contains an archive of free-to-access editorials in many Nature journals that discuss aspects of the peer-review system and process.

Posted by Maxine Clarke on January 29, 2009
Categories: Systems | Permalink | Comments (4)

"Scientific publishing depends on expert peer reviewers. Instead of perpetually arguing about the reliability and fairness of peer review, authors, editors and referees should seek to optimize this time-tested system." So opens the January editorial of Nature Neuroscience (12, 1; 2009).
The editorial discusses the media reporting of a ferocious argument about the merits of a paper published in Cell , and a subsequent blog debate hosted by The Scientist. The controversy between scientists in this discipline concerning this paper "has again ignited a debate on the flaws of editor-managed anonymous peer review", write the Nature Neuroscience editors. "We maintain, however, that despite occasional unfortunate lapses, anonymous peer review remains the best quality-control process that we have." The editorial goes on to discuss how journals can best optimize the process.
In Nature News this week (457, 245; 15 January 2009) another technical dispute is discussed, this time concerning a widely circulating preprint attacking much of the published research in social neuroscience involving functional magnetic resonance imaging (fMRI). This affair is exacerbated by the fact that the circulating preprint is not due to be published, with responses from the authors of some of the criticized studies, for another nine months, according to the Nature News story. The criticisms have already been covered in publications such as Newsweek, as well as the inevitable blogosphere outpourings - and at least some of the criticized authors say the first they heard of the preprint was when they were contacted by journalists.
How scientific reports should be peer-reviewed and, after publication, scrutinized are matters that are decided upon by the journals, their editors, and their publishers and/or societies - in the case of articles submitted to and published in the Nature journals, the peer-review process is described here, and the post-publication corrections process here. Good journals have processes for investigating technical criticisms and complaints about the papers they publish. Nature's, for example, is here. Often, a resolution is not clear-cut at the outset, when the complaint is first received by the journal, however clear it may be in the mind of the complainer. A proper outcome depends on independent peer-reviewers, as well as editors, examining the complaint together with a measured response from the study's authors. Playing out such investigations in the kangaroo courts of the popular press, or in unfettered comments on the Internet between people who have been described as "recreationally outraged", not only obscures logical, technically informed investigation, but unnecessarily exacerbates emotions and arguments so that, in the end, all that is remembered is the heat - not any light.

Posted by Maxine Clarke on January 16, 2009
Categories: Ethics, Quality and value, Systems | Permalink | Comments (2)

Via press release, The EMBO Journal will be publishing online author and referee comments from this year (2009). “The EMBO Journal has been our flagship publication for 27 years, sharing knowledge broadly within the molecular life sciences community,” said Hermann Bujard, director of EMBO (European Molecular Biology Organisation). “We are excited by the editorial changes that will make publication of research findings more transparent, complete and visible.”
By making the evaluation of manuscripts visible to everyone, The EMBO Journal aims to encourage constructive referee and author argumentation. Younger scientists will gain valuable insight into how to publish their research findings as well as how to deal with criticism.
"The EMBO Journal has an efficient and reasonable editorial process,” writes Executive Editor Pernille Rørth in an advanced online publication of the editorial in the first issue for 2009 (EMBO J. 28 , published online 4 December 2008). “A transparent editorial process will help demystify decisions.”
Beginning with manuscripts submitted in 2009, a supplementary process file will be included with the online publication of papers. This file will show all dates relevant to manuscript processing and communications between the author, editors, referees and comments to the decision letter. Readers will learn about why referees find the paper interesting, any gaps they may have identified in the initial research findings, and how the gaps were resolved in revision. Referee identities will be anonymous and confidential comments between the referee and editors will remain so. Authors will have the option to decline publication of the editorial process when they submit manuscripts, but are encouraged to participate.
Instructions for authors are available here.

Posted by Maxine Clarke on January 05, 2009
Categories: Journals, Systems | Permalink | Comments (1)

The US National Institutes of Health (NIH) will introduce new guidelines in January 2009 allowing biomedical researchers to amend and resubmit a failed funding application only once, as part of an overhaul of the peer-review system for evaluating grant proosals (see Nature 455, 841; 2008). Applicants whose grants are unfunded after the second submission may reapply only after designing a new proposal. NIH has previously suggested not allowing resubmissions, but decided against this step after an outcry from researchers (see Nature 453, 835; 2008).
NIH estimates that the move will reduce the number of applications by up to 5,000 — welcome news as it struggles to evaluate about 55,000 applications this year. In 2007, only about 30% of awards were granted to first-time submissions.
According to a comment to the Nature News story by Jeremy Green, in the current UK funding system, the research councils and the Wellcome Trust operate a no re-submission policy, although depending on the subject area an applicant might be able to submit a proposal rejected by one funder to the others. However, applicants to the research councils do get an opportunity before the grant review panel sits to see and respond to the reviewers' comments.

Posted by Maxine Clarke on November 14, 2008
Categories: Systems | Permalink | Comments (0)

A News story in the 12 June issue of Nature (453, 823; 2008) by Meredith Wadman:
Responding to hundreds of critical comments, the US National Institutes of Health (NIH) has reversed several controversial proposals made in February as part of a year-long effort to overhaul the agency's peer-review system (see Nature 451, 1035; 2008).
As part of an initiative called Enhancing Peer Review, announced in a finalized form on 6 June, the agency will spend at least $200 million annually over the next five years to foster groundbreaking, investigator-initiated research. Of that, at least $250 million will go to a new beast: a Transformative R01 Award, a reach-for-the-skies version of the NIH's basic grant. The remaining $750 million will go to existing awards that reward risk and innovation: the Eureka, New Innovator and Pioneer awards.
The changes “are concrete solutions that will maximize flexibility, remove any unnecessary burden, stimulate new innovation and promote transformative research”, says NIH director Elias Zerhouni.
They include rewards for long-serving reviewers; a streamlined, 12-page R01 grant application, down from 25; and a seven-point, integer scoring scale for grant applications, which will be assessed across five criteria: impact, investigators, innovation, feasibility and environment. Current applications are graded on a 41 point scale, from 1.0 to 5.0, raising complaints that they claim a degree of accuracy that can't be scientifically defended.
Among the controversial proposals shelved by the agency was a recommendation that all applications, even those on a second or third submission, would be treated as new, without reviewer access to prior reviews.
Gone, too, is the category “not recommended for resubmission”, which had been suggested for dismal applications. Scientists felt that branding projects with “a clear, checkbox-driven stigma is bad, that it could have unintended consequences”, Jeremy Berg, director of the National Institute of General Medical Sciences, told the advisory committee.
Berg and Lawrence Tabak, director of the NIH's dental institute, head the group that developed the recommendations and are charged with implementing them over the next 18 months.
The agency also jettisoned a “minimum effort requirement” that would have required principal investigators to commit at least 20% of their time to any single NIH grant — an item of particular concern for 'grandee grantees' (see Nature 452, 258–259; 2008). Instead, grantees will need to indicate if they will have more than $1 million in cumulative NIH funding.

Posted by Maxine Clarke on June 12, 2008
Categories: Grants, tenure and positions, Systems | Permalink | Comments (2)

The Editorial 'Working double-blind' (Nature 451, 605–606; 2008), also republished on this blog and stimulating more than 70 comments, referred to a study (1) that found more female first-author papers were published using a double-blind, rather than a single-blind, peer-review system. The data reported in ref. 1 have now been re-examined (2). The conclusion of ref. 1, that Behavioral Ecology published more papers with female first authors after switching to a double-blind peer-review system, is not in dispute. However, ref. 2 reports that other similar ecology journals that have single-blind peer-review systems also increased in female first-author papers over the same time period. After re-examining the analyses, Nature has concluded that ref. 1 can no longer be said to offer compelling evidence of a role for gender bias in single-blind peer review. In addition, upon closer examination of the papers listed in PubMed on gender bias and peer review, we cannot find other strong studies that support this claim. Thus, we no longer stand by the statement in the fourth paragraph of the Editorial, that double-blind peer review reduces bias against authors with female first names.
References
1. Budden, A. E. et al . Trends Ecol. Evol. 23, 4–6 (2008).
2. Webb, T. J. , O'Hara, B. & Freckleton, R. P. Trends Ecol. Evol. doi:10.1016/j.tree.2008.03.003 (2008).

Posted by Maxine Clarke on June 05, 2008
Categories: Ethics, Quality and value, Systems | Permalink | Comments (2)

In the latest Nature Reports Stem Cells Inside the Paper feature, senior Nature editor Natalie DeWitt discusses the paper by H. H. Chang et al. 'Transcriptome-wide noise controls lineage choice in mammalian progenitor cells', published in Nature 453, 544-547 (2008).

The Editor's summary of this paper: Even in clonal populations of cells, there is significant phenotypic variation from cell to cell. This could reflect the 'noise' inherent in gene expression: or the various cell states could represent stable phenotypic variants. Chang et al. analysed the behaviour of an 'outlier' in clonal populations of mouse haematoipoietic stem cells that had very high expressions of the stem cell marker Sca-1 and found that outliers possessed distinct transcriptomes. Though the transcriptomes eventually reverted back to that of the median cells, while they differed they could drive the cells to express characteristics of distinct cell fates. Thus clonal heterogeneity of gene expression may not be due to noise in the expression of individual genes, but rather is a manifestation of metastable states of a slowly fluctuating transcriptome. These fluctuations may govern the reversible, stochastic priming of multipotent progenitor cells in cell fate decision.

The Nature Reports Stem Cells article provides the details of what the peer-reviewers thought of this paper when it was submitted, and how the authors responded. The initial view of the three peer-reviewers can be summarized as follows:

Reviewer 1 In summary, we believe that this paper reports a novel and important biological mechanism for differentiation: it gives evidence that slow stochastic variations in stem cell state last for a long time and result in different fates. It relates the recent finding of slow fluctuations in human protein levels to the biological outcome of cell fate, and finds long lasting differences in transcriptomes in different subpopulations. It is an excellent choice for Nature, provided that the comments are addressed.

Reviewer 2 Although the phenomenon described is immensely interesting and the idea of heterogeneity being retained within even clonal populations of cells is plausible, the authors merely describe this phenomenon and in some instances, do not provide conclusive data to support their interpretation. If a mechanism was determined this would definitely aid in its novelty and interest. In its current form, I believe this manuscript should not be accepted at Nature.

Reviewer 3 This is a very timely and interesting paper, that should be of interest to a broad range of researchers. However, I have some serious concerns over the modeling.

The details of the reviewers' concerns, and the authors' response, can be read at Nature Reports Stem Cells,

You can also read a set of question-and-answer sessions between editor Monya Baker and several of the peer-reviewers of the recent Nature Insight on Regenerative Medicine, here on The Niche, the blog of Nature Reports Stem Cells.

Posted by Maxine Clarke on June 02, 2008
Categories: Systems | Permalink | Comments (0)

Nature Neuroscience is joining a consortium of journals that enables reviews to be transferred from one journal to another, while allowing authors, referees and editors to control their degree of participation in the system flexibly. The reasons for the decision are explained in this month's (April) Editorial (Nat. Neurosci. 11, 375; 2008). Briefly, in January, a group of editors, supported by the Society for Neuroscience, implemented a system for transfer of submitted manuscripts between journals that allows voluntary participation by authors, referees and editors, known as the Neuroscience Peer Review Consortium. This consortium reduces the overall reviewing workload of the community by allowing authors to continue the initial review process when their paper moves from one consortium journal to another, once the paper has been rejected or withdrawn from the first journal. This arrangement is similar to the manuscript transfer system that has been available within the Nature family of journals , and all the other journals published by Nature Publishing Group, for almost a decade.
The neuroscience transfer system, described in detail in the Editorial and at the Neuroscience Peer Review Consortium website, is voluntary for authors and peer-reviewers. Journal editors have full discretion in deciding how to use transferred reviews. The receiving editor may choose to accept or reject a paper based on these reviews, without further consideration; to send the paper to some or all of the previous referees for evaluation of the authors' revisions; or to request a fresh set of reviews from new referees. Only comments to the authors are transferred to the receiving journal. Confidential comments to the editors are not passed along. Thus, to ensure transparency in the review process, both at Nature Neuroscience and at other journals after the paper has been transferred, journal editors encourage referees to include all their concerns about the paper in comments to the authors. According to the Nature Neuroscience Editorial, the small amount of extra time required to phrase comments diplomatically for the authors should be more than counterbalanced by the resulting improvement in the peer-review process.
Many members of the community have strong views on the issue of confidential comments, which can be found on the Action Potential blog, and which has previously been discussed at Peer to Peer. The success of the neuroscience journals' transfer system will be evaluated at the end of this year. Nature Neuroscience editors will also be evaluating the journal's participation on an ongoing basis, so they encourage authors, referees and readers to share their comments with the editors, either on Action Potential blog or privately by e-mail.

Posted by Maxine Clarke on March 31, 2008
Categories: Ethics, Quality and value, Systems | Permalink | Comments (0)

The newly launched journal Scholarly Research Exchange, published by Hindawi, is for original research articles in all areas of science, technology and medicine. The journal is operating a transparent peer-review system, in which authors and reviewers interact directly throughout the peer-review process. Authors submitting to Scholarly Research Exchange suggest potential reviewers, who are then approved by the journal's editors. Reviewers are asked to provide an assessment of the quality of the manuscript, a written critique for the authors, and a written commentary for the journal's readers. When the journal accepts its first articles for publication, the reviewers' commentaries and their assessment of the manuscript's quality will be published with them. In addition to these reviewers' evaluations, members of the scientific community will be able to participate in a discussion around every manuscript.
A journal with a similar, but not identical, peer-review process is Biology Direct, published by BioMed Central, currently covering the areas of genomics, bioinformatics and systems biology, immunology, and mathematical biology. Biology Direct makes the author responsible for obtaining reviewers' reports, via the journal's editorial board, operates an open peer-review process, and publishes reviewers' reports with the articles. It has been publishing articles since January 2006.

Posted by Maxine Clarke on March 12, 2008
Categories: Systems | Permalink | Comments (0)

How many papers does the typical researcher review in a year? How long do they take? And why do they do it? For the answers, read the Editorial in this month's (March) issue of Nature Nanotechnology, "Who'd be a referee?" (3, 119; 2008). The Editorial reports some of the findings of the recent Publishing Research Consortium report on peer-review (previously discussed at Peer to Peer), whose survey revealed that the "average review takes about 8.6 hours (with a median of about 5 hours) and is completed within 3–4 weeks, although there are significant differences between the four broad subject areas covered by the survey: in the physical sciences and engineering, for instance, the average (mean) is 10.4 hours, compared with 6.3 for clinical researchers. Of course, on top of this, many researchers spend considerable time reviewing grant applications for funding agencies, which can be equally onerous and possibly even more important than reviewing papers."
For similar statistics, and a more general discussion of the peer-review process in publishing, see the Nature Nanotechnology Editorial, or see here for Nature Cell Biology's take.

Posted by Maxine Clarke on March 10, 2008
Categories: Quality and value, Systems | Permalink | Comments (0)

This month's (March 2008) free-access Editorial in Nature Cell Biology (10, 247; 2008) addresses the journal's peer-review process: specifically, what the journal does to ensure that its selection process is fair.
From the Editorial:
"A legitimate question for editors at Nature Cell Biology is whether we are considering alternatives to the status quo of anonymous peer review, in particular, why we do not use a 'double blind' process (which received significant endorsement in a recent survey of the Publishing Research Consortium). The existing process, based on a thorough pre-selection by five full-time editors and subsequent external peer review by carefully selected referees, works well — individual stories of woe notwithstanding."
The editorial goes on to outline how changes to the system, when being considered, must show a demonstrable improvement to the process. Several of the commonly proposed alternatives to the 'single blind' system are discussed in this light (see here for a recent popular debate on the topic at this blog). The Editorial concludes with a summary of the process as currently run by the Nature Cell Biology editors, together with the journal's planned enhancments.
The editors welcome your views on the Editorial as comments to this post.
Further information about the Nature journals' peer-review policies are available at the authors' and reviewers' website.
Connotea tags for peer-review.

Posted by Maxine Clarke on March 06, 2008
Categories: Journals, Systems | Permalink | Comments (0)

This is the text of an editorial published in Nature in the 7 February issue (Nature 451; 605-606; 2008) . We, the journal editors, welcome your comments and suggestions.
Corrected 4 June 2008: Nature 453, 711 (2008).
Please see end of this post for text of correction correction to this editorial*
Should there be author anonymity in peer review?

Double-blind peer review, in which both authors and referees are anonymous, is apparently much revered, if not much practised. The Publishing Research Consortium (PRC) has assessed attitudes towards peer review among 3,000 academics in an international survey across the sciences and humanities. The results, released last month, strongly affirm the value of peer review [See earlier posting on Peer to Peer.]. They also highlight that 71% have confidence in double-blind peer review and that 56% prefer it to other forms of review. Support is highest with those who have experienced it (the humanities and social sciences) or where it is perceived to do the most good (among female authors). The least enthusiastic group is editors. So is it time for editors, and those at Nature in particular, to reconsider their position?
If referees know the authors' identities, it may leave the latter vulnerable to biases about them or their previous work, their gender, their nationality or their being new to an area of research. But the PRC survey supports the contention of Nature and others that identifying authors stimulates referees to ask appropriate questions (for example, differentiating between a muddy technical explanation and poor experimental technique). Knowing author identities also makes it easier to compare the new manuscript with the authors' previously published work, to ensure that a true advance is being reported. And knowing rather than guessing the identities of authors encourages reviewers to raise potential conflicts of interest to the editors.
Is there evidence that double-blind peer review presents a better alternative? It would do so if it generated more constructive comments in the minds of editors and authors, or if the identity of authors were truly protected, or if biases were reduced. So far, the jury is out. Although at least one study in the biomedical literature has suggested that double-blind peer review increases the quality of reviews, a larger study of seven medical journals (refs 2, 3) indicated that neither authors nor editors found significant difference in the quality of comments when both referees and authors were blinded. Referees could identify at least one of the authors on about 40% of the papers, undermining the raison d'être for double-blinding. The editors at the Public Library of Science abandoned double-blind peer review because too few requested it and authors were too readily identified.
The one bright light in favour of double-blind peer review is the measured reduction in bias against authors with female first names (shown in numerous studies, such as ref. 4 [previously discussed on Peer to Peer]). This suggests that authors submitting papers to traditionally minded journals should include the given names of authors only on the final, published version.
The double-blind approach is predicated on a culture in which manuscripts-in-progress are kept secret. This is true for the most part in the life sciences. But some physical sciences, such as high-energy physics, share preprints extensively through arXiv, an online repository. Thus, double-blind peer review is at odds with another 'force for good' in the academic world: the open sharing of information. The PRC survey found that highly competitive fields (such as neuroscience) or those with larger commercial or applied interests (such as materials science and chemical engineering) were the most enthusiastic about double-blinding, whereas fields with more of a tradition for openness (astronomy and mathematics) were decidedly less supportive.
Where does this leave journals? Editors have the responsibility to provide a neutral bridge between referees and authors and so may help to better shield authors from bias. Easily said! The evidence of the PRC survey suggests little faith in that impartiality, but editors — certainly at Nature and its related journals — take that responsibility seriously.
Nature's policies over the years have generally moved towards greater transparency. Coupling that with the lack of evidence that double-anonymity is beneficial makes this journal resistant to adopting it as the default refereeing policy any time soon. But many of our readers are referees as well as authors. We welcome their views on author anonymity from both vantage points.
1. Publishing Research Consortium Peer Review in Scholarly Journals (Mark Ware Consulting, Bristol, 2008).
2. Justice, A. C. et al. J. Am. Med. Assoc. 280, 240–242 (1998).
3. Cho, M. K. et al. J. Am. Med. Assoc. 280, 243–245 (1998).
4. Budden, A. E. et al. Trends Ecol. Evol. 23, 4–6 (2008).
The Nature journals' peer-review policies are described at our Author and Reviewers' website, where you can also find links to our editorials on the topic, all free to access.

CORRECTION to the editorial:
The Editorial 'Working double-blind' (Nature 451, 605–606; 2008) referred to a study1 that found more female first-author papers were published using a double-blind, rather than a single-blind, peer-review system. The data reported in ref. 1 have now been re-examined2. The conclusion of ref. 1, that Behavioral Ecology published more papers with female first authors after switching to a double-blind peer-review system, is not in dispute. However, ref. 2 reports that other similar ecology journals that have single-blind peer-review systems also increased in female first-author papers over the same time period. After re-examining the analyses, Nature has concluded that ref. 1 can no longer be said to offer compelling evidence of a role for gender bias in single-blind peer review. In addition, upon closer examination of the papers listed in PubMed on gender bias and peer review, we cannot find other strong studies that support this claim. Thus, we no longer stand by the statement in the fourth paragraph of the Editorial, that double-blind peer review reduces bias against authors with female first names.
References
Budden, A. E. et al . Trends Ecol. Evol. 23, 4–6 (2008).
Webb, T. J. , O'Hara, B. & Freckleton, R. P. Trends Ecol. Evol. doi:10.1016/j.tree.2008.03.003 (2008).

Posted by Maxine Clarke on February 06, 2008
Categories: Systems | Permalink | Comments (85)

Much attention is being given to a Commentary in the current issue of Nature (Nature 451, 379-399; 2008), A tale of two citations, some of which I have attempted to encapsulate in this Nautilus post, for those interested. Although the issues immediately concern a possible increase in duplicate publication and plagiarism, as detected by software systems and database searches, peer reviewers are an integral part of the check/balance procedures that journals use. For this reason, I thought it well worth highlighting here the comment by Brian Derby at the Nature Network forum currently discussing these questions. Part of Dr Derby's response:

"As a referee I have identified duplicate or severely overlapping content while reviewing papers in the past (for reasonably high profile/impact factor journals). I do not search for duplication routinely but, as someone who is used to referee papers in particular niche areas, I received both papers in one instance and in another I had read an on-line pre-pub before receiving the duplicate. The authors will not be named as that would break referee confidentiality but they were from well known institutions in the developed world.
What was the common factor (apart from the paper!) – the authors were relatively junior new appointments. Younger academics seem to feel themselves under a lot of pressure to publish. In my department I believe that my younger colleagues are much more sensitive to impact factor than is possibly healthy when they consider where to publish an article."

Posted by Maxine Clarke on January 28, 2008
Categories: Ethics, Quality and value, Systems | Permalink | Comments (1)

The Publishing Research Consortium publishes a study this month (January 2008) in whch more than 3,000 senior authors, reviewers and editors were asked about the peer-review system. The conclusions are that researchers want to "improve, not change, the system of peer review for journal articles". According to the report, a summary of which is available (1.7 MB; PDF), more than 93 per cent of respondents believe that peer review is necessary, and more than 85 per cent say that it helps to improve scientific communications and increases the overall quality of published papers.
Although many respondents pointed out the operational difficulties in double-blind peer review, two-thirds of respondents felt that it is the most objectively fair system, compared with single-blind (the current prevalent system). Alternatives such as post-publication and open peer-review were not popular.
While of the majority of respondents saw peer review as an effective filter for research, some did not think it was effective at detecting plagiarism, fraud or misconduct. Interestingly, most reviewers among the respondents thought that paying peer-reviewers would be too expensive for publishers; most of them said that they perform reviewing as part of their support to their research community.
The full report is available here (1 GB; PDF). According to the Publishing Research Consortium, the main objective of the study was "to measure the attitudes and behaviour of the academic community with regard to peer review. This will inform debate concerning peer review, and underpin discussions, either in discussion lists or at future workshops/conferences."
This new report comes as the NIH (National Institutes of Health) finish analysing the thousands of responses to their assessment of grant peer review. Lawrence Tabak and colleagues are filtering the list into a set of key recommendations, which will be given to Elias Zerhouni, director of NIH, at the end of February.
Update, 29 Jan 2008. Nature Neuroscience discusses the NIH peer-review exercise in its February issue Editorial (Rethinking grant review Nature Neuroscience 11, 119; 2008).

Posted by Maxine Clarke on January 25, 2008
Categories: Quality and value, Systems | Permalink | Comments (0)

Double-blind peer review, in which neither author nor reviewer identity are revealed, was introduced by the journal Behavioral Ecology in 2001. Amber E. Budden et al., in an article published in Trends in Ecology and Evolution this month (Trends Ecol. Evol. 23, 4-6; 2008) report "a significant increase in female first-authored papers" compared with a similar journal, Behavioral Ecology and Sociobiology. From the authors' conclusions:
"A difference of 7.9% in the proportion of female first-authored papers following the implementation of double-blind review in BE is three times greater than the recorded increase in female ecology graduates in the USA across the same time period and represents a 33% increase in the representation of female authors. Furthermore, this increased representation of female authors more accurately reflects the (US) life sciences academic workforce composition, which is 37% female.
The consequences of this shift could extend beyond publications. If females are less successful in publishing research on account of their gender, then given the current practices associated with appointment and tenure, and the need for women dramatically to out-compete their male counterparts to be perceived as equal [C. Wenneras and A. Wold, Nepotism and sexism in peer-review, Nature 387 341–343; 1997] any such publication bias impedes the progress of women to more advanced professional stages."

Posted by Maxine Clarke on January 21, 2008
Categories: Ethics, Systems | Permalink | Comments (1)

In his Nature Network summary of a recent publishing meeting, Yiorgos Apidianakis describes his opinion that a peer-review "score" is a more desirable (and efficient) indicator of scientific excellence than the currently used impact factor of the journal that publishes the work. Charles G. Jennings's wrote, as part of Nature's peer-review debate, “It is common to bemoan the over-reliance on quantitative markers such as impact factors for assessing scientists’ abilities (and indeed there is much to bemoan), but until committee members have time to read every paper on every applicant’s CV, they will have to rely at least in part on proxy indicators.” Dr Apidianakis believes that an ideal indicator would be a score from a unified peer-reviewing system, or a central agency "that will thoroughly, rigorously and objectively evaluate any given work to be published, using again specialized scientists as reviewers. Having an evaluation score from such an agency, scientists can include this score to their publication record."

Dr Apidianakis thinks that such a system could work in practice, using the example of the NIH evaluation scoring system -- while admitting it would be very expensive. Irrespective of this practical obstacle (as well as others) I wonder how the quality of the agency's "score" can be standardized? The establishment of, in effect, one giant journal of research, with all the world's scientists signed up to it as peer-reviewers, is a stimulating concept. But, given the many divergences of views within fields, how could a centralized scoring system work? The IPCC has attempted a similar kind of approach for one discipline, climate change -- even though most scientists in the field broadly agree with the IPCC's assessment of research output, this consensus requires massive bureaucratic baggage, including many international meetings and vast reports justifying decisions. Yet there is a substantial minority of scientific dissentors, and many members of the public are sceptical of a unified approach resulting in "science by endorsement".

The current peer-review system works very well "in miniature", whether as operated by those journals able to call on the most thoughtful scientists in a field who do the most cutting edge research, or by literature reviews or research ranking services, usually written or operated by one or a few individuals. Scientific research itself has benefited from large author collaborations across the whole spectrum of disciplines, from astronomy and nuclear physics to genomics and cell signalling, facing the challenge of making sense of vast quantities of data. Are there the same intellectual and innovative advantages to be gained by a single-managed peer-review system for all of science?

Posted by Maxine Clarke on December 04, 2007
Categories: Quality and value, Systems | Permalink | Comments (0)

Dr Robert Higgs of the Independent Institute, writes:

Journalists, politicians and advocacy groups refer to “peer-reviewed research” and “scientific consensus” as the authoritative last words on controversial matters involving the natural sciences, from climate change to stem-cell research and genetically engineered foods. But many people have an unrealistic view of how the scientific community actually works.
The peer-review process is not, contrary to popular belief, a nearly flawless system of Olympian scrutiny. Any editor of a peer-reviewed journal who desires to reject or accept a submission can easily do so by choosing appropriate referees.
Unfortunately, personal vendettas, ideological conflicts, professional jealousies, methodological disagreements, sheer self-promotion and irresponsibility are as much part of the scientific world as any other. Peer review cannot ensure that research is correct in its procedures and conclusions. A part of the work in every discipline – from the physical sciences to economics –consists of correcting previous mistakes.
At any given time, “scientific consensus” may exist about various matters. Over time, however, new interpretations, tests or observations may demolish that consensus. For instance, in the mid-1970s, an apparent scientific consensus existed that our planet was about to enter another Ice Age. Drastic proposals, such as exploding hydrogen bombs over polar icecaps to melt them. and damming the Bering Strait to prevent icy waters from entering the Pacific, were put forth by reputable scientists and seriously considered by the US government.
The truth is that scientific research at the upper echelons occurs within a fairly small world. Leading researchers attend the same conferences, belong to the same societies, review one another’s work for funding organizations, and so forth. If you do not belong to this tight fraternity, it becomes extremely difficult to gain a hearing for your work, to publish in a “top” journal, to acquire a government grant, to receive an invitation to participate in a scientific conference, or even to place your grad students in decent positions.
“Scientific consensus” often emerges because the members of this exclusive club, and those who support them, have too much invested in the reigning ideas to let go. In this context, it behooves bright young scientists not to rock the boat by challenging anything fundamental or dear to the hearts of those who constitute review committees of funders or journals. The terms "peer review" and "scientific consensus" often serve to suggest a process of disinterested neutrality and saintly pursuit of truth. Like every other human endeavour, however, science is conducted by people with the full range of human emotions and motives.
Good rules of thumb for the non-scientist might be the following: government-funded research that is used to justify that government’s policy should be suspect, whether or not it’s peer-reviewed; and the research of scientists who appear at press conferences in the company of politicians or activists whose agendas they are there to support should be suspect, whether or not the work upholds the consensus opinion.

Robert Higgs is Senior Fellow in Political Economy at the Independent Institute, editor of the quarterly journal The Independent Review, and the author of Depression War and Cold War, as well as numerous books and more than 100 articles in scholarly journals.

Posted by Maxine Clarke on September 17, 2007
Categories: Quality and value, Systems | Permalink | Comments (5)

Todd A. Gibson of the University of Colorado writes in Nature's Correspondence (Nature 448, 408; 2007) pages:

Shi V. Liu's correspondence, 'Why are people reluctant to join in open review?' (Nature 447, 1052; 2007; see also Peer to Peer discussion), struck a chord. Recently, I stayed my hand before submitting a comment to an online article, because the comment included as-yet unpublished research that I was reluctant to reveal in such a forum.
I believe that there are two ways to encourage online commenting. These would require little additional commitment, but would improve journal quality and enhance the development of review skills among young scientists.
First, journals could institute periodic post-publication review, in which the journal would solicit formal review of the article, focusing on how well its methods and results have held up, given the research that has been published in the intervening period. Such reviews would provide valuable historical perspective. Second, young scientists participating in journal clubs could be asked to derive and post a consensus comment on the article under discussion.

Posted by Maxine Clarke on August 09, 2007
Categories: Mentoring and training, Systems | Permalink | Comments (0)

Mario of Mario's Entangled Bank blog discusses an article in Wired magazine about Nature Precedings. Mario higlights Wired's criticisms of Precedings for its "controversial" title, and for offering to upload scientists' "effluvia and detritus" that would never appear in a journal. To the contrary, he writes, the site (on which he has been a beta tester) states that it "provides a rapid way to disseminate emerging results and new theories, solicit opinions, and record the provenance of ideas." From Mario's post:

"In other words, contrary to detritus and effluvia, Nature Precedings will host research at the cutting edge, some of which may have errors in it that will be "corrected" with the collaborative efforts of the research bazaar. Most of the posted research will, however, be correct (after all, it is your reputation that is at stake) and it's early pre-print disemination will only be of benefit to the scientific community at large. This basically also addresses the last paragraph, "There's also a danger of errors being missed that would have been picked up in the peer review process". On the contrary, the recent PLoS retraction (see Show me the code and Exemplary retraction of high profile paper) confirms what we all academic researchers already know, that the peer review process is far from being fool proof. A forum like Nature Precedings, or arXiv for that matter, aid in the peer review process by disseminating pre-print cutting edge research findings thus allowing errors to be caught early on."

Posted by Maxine Clarke on July 20, 2007
Categories: Systems | Permalink | Comments (0)

From Nature's Correspondence section this week (Nature, 447, 1052; 28 June 2007)
Sir -- I was excited when Nature launched its trial for open peer review last year, but disappointed by the outcome . I have also been tracking the progress of another open review journal, Biology Direct (see article in Nature's peer-review debate ). Even after a high-profile launch with guaranteed indexing by PubMed, this journal has published only 52 articles and received only two comments over 16 months.
In contrast to Biology Direct, another journal that offers open commenting, PLoS One , has published 1,189 articles in its first six months. But has PLoS One achieved its stated goal of post-publication open comments? I find that even the 'most annotated' category of articles usually receives just a few comments. The journal has recently replaced its 'most annotated' with a 'recently annotated' category. A check of all 'recently annotated' articles demonstrates that their commenting rates are low (zero or just a few), even for articles that are likely to have broad appeal and/or are in 'hot' research areas.
Why is there a general lack of interest among the scientific community in open commenting on submitted or published papers? I believe there are two main reasons. First, participation does not earn any tangible credit or benefit for the reviewers and commentators. Second, publicly critical comments are a risk for those who make them.
Shi V. Liu
Scientific Ethics, Apex, North Carolina

[Correction added by Maxine Clarke: Chris Surridge of PLOS One points out in a comment to the post that the journal published just over 550 articles in its first six months, not 1,189. Thank you, Chris.]

Posted by Maxine Clarke on June 29, 2007
Categories: Systems | Permalink | Comments (2)

Among the excellent content on Nature Reports Stem Cells is an especially exciting regular feature, "Inside the Paper", which zeros in on recent scientific papers in the field published in Nature and provides detailed expert comment and author responses drawn from the peer-review process. The editors intend this feature to let readers see the paper’s context, strengths and caveats, as well as make the peer- review process a little more transparent.

Two Inside the Paper features have been published this month. The first, Cloning from Chromosomes, discusses the Article by D. Egli et al., Developmental reprogramming after chromosome transfer into mitotic mouse zygotes, in Nature 447, 679–685 (2007). Read a panel of experts' comments on this interesting article, responses from the authors, and comment yourself on The Niche (the Nature Reports Stem Cells blog).
The second Inside the Paper (21 June), Rewriting in blood, is about a paper by I. M. Samokhvalov et al(Nature 446, 1056–1061; 2007), reporting a surprising origin for blood stem cells. As before, you can read some of the peer reviewer's comments, responses from the authors and invite you to add your contribution on The Niche.

Posted by Maxine Clarke on June 28, 2007
Categories: Quality and value, Systems | Permalink | Comments (0)

Nature Precedings is now out of "beta testing" and is launched. This new community service is described at Nautilus, the NPG blog for present and future authors. Submissions are screened by our professional curation team for relevance and quality, but are not subjected to peer review. High-quality contributions from biology, medicine (except clinical trials), chemistry and the Earth sciences are welcomed.

More details about Nature Precedings can be found here. Because they have not been peer-reviewed, many of the findings you read at Nature Precedings may be preliminary or speculative, and remain to be confirmed. Please bear this in mind when deciding how seriously to take them.

Submissions are not accepted from fields in the physical sciences that are are already well served by preprint servers such as arXiv.org. Content that considered to be non-scientific or pseudoscientific is rejected. We accept only genuine contributions from qualified scientists. This will usually require submitters to have a recognized academic affiliation. Incomplete submissions will also be rejected. This is a free service, so please help us to help you by completing all relevant sections of the submission form.

The reactions of scientists and publications to the launch of Nature Precedings can be found at this Connotea page, which is regularly updated. Here are one or two such articles:
Nature Precedings pre-print server for biomedical research
Nature Precedings: A nicer version of ArXiv[e] for biomedical research
Chemistry Central: A new preprint server from Nature.

Posted by Maxine Clarke on June 18, 2007
Categories: Open peer review, Systems, Technical solutions | Permalink | Comments (0)

Steven Novella of the blog NeuroLogica writes an essay on peer review. " Peer-review is a critical part of the functioning of the scientific community, of quality control, and the self corrective nature of science", he writes. "But it is no panacea. It is helpful to understand what it is, and what it isn’t, its uses and abuses."
The essay is a short but informative account of the standard peer-review processes of many scientific journals, including a short paragraph about post-publication peer-review. At the time of writing this post, there are two comments on the subject of peer-review and fraud.
Dr Novella's "bottom line: peer-review is a necessary component of quality control in science, but is no guarantee of quality, and you have to know the details of the journal that is providing the peer-review."

Posted by Maxine Clarke on June 14, 2007
Categories: Systems | Permalink | Comments (0)

See Nature Reports Stem Cells for a fresh perspective on a paper reporting a technique for cloning from zygotes. In this fortnight’s Inside the Paper, a new form of scientific reporting pioneered by Nature Reports Stem Cells, you can read a moderated discussion between the authors and the paper's peer-reviewers. Readers can learn what the foremost experts in the field had to say about the submitted paper’s strengths and shortcomings. See what the authors saw, and read their responses as they revised their paper for eventual publication in Nature. You can also add your own comments on The Niche, the Nature Reports Stem Cells blog.
Featured paper: D. Egli et al. Developmental reprogramming after chromosome transfer into mitotic mouse zygotes. Nature 447, 679–685 (2007).

Posted by Maxine Clarke on June 13, 2007
Categories: Systems, Technical solutions | Permalink | Comments (0)

Picking up on the Nature Structural and Molecular Biology Editorial "New data at conferences, please", discussed here last week, Mario Pineda-Krch writes on his blog (Mario's Entangled Bank) about how the idea of the conference as a type of open peer-review process reminds him of the 1997 essay The academic Cathedral and the Bazaar by Eric S. Raymond.

In the Cathedral model the research process occurs behind closed doors among a exclusive group of researchers where progress is reported in peer-reviewed publications. In the Bazaar model the research is conducted in full view of the public......The basic tenet of the Bazaar model has in the field of software development termed Linus' law, i.e. "given enough eyeballs, all bugs are shallow". In an academic research environment the pros of the Bazaar model hinge on the fact that the more widely available the your research methodology and results is for public testing, scrutiny, and experimentation, the more rapidly all forms of errors and omissions will be discovered. In contrast, in the Cathedral model an inordinate amount of time and energy must be spent hunting for errors due to the limited number of eyeballs.

Mario, a postdoctoral researcher who works on ecological population dynamics at the University of California, Davis, believes that "academia would be a better place if more people would embrace this type of openness. Maybe the conferences setting could be the natural stage where the Bazaar movement could start."

Posted by Maxine Clarke on June 12, 2007
Categories: Systems | Permalink | Comments (0)

Inherently Responsive is the blog of the journal Heredity, published by NPG. As explained at the post Your chance to respond, the journal is seeking to improve its peer-review process using the opportunities provided by the web. One proposed model is to publish papers accompanied by referees’ comments, another is to publish correspondence columns in the journal after publication of the paper, to air legitimate differences of scientific opinion.
Inherently Responsive combines aspects of these two approaches. It publishes rapid feedback on papers that have appeared in Heredity: technical comments, relevant work published elsewhere and so on. Here is one example, about inheritance of litter size in Arctic foxes. It also plans to publish comments provided for public consumption by the peer-reviewers, so readers can see why the paper is controversial or why publication was recommended.
Inherently Responsive also features discussion on the editorial direction of the journal. In large part the content of Heredity is determined by what is submitted, but some of it is are commissioned. The blog seeks to obtain journal readers' views on whether important areas are being negelected.

Posted by Maxine Clarke on June 12, 2007
Categories: Systems | Permalink | Comments (0)

NIH Establishes Working Groups to Examine Peer Review

The Director of the US National Institutes of Health (NIH), Dr Elias A. Zerhouni, annnounced on Friday last week (8 June) the formation of two working groups to examine the NIH peer review process, with the goal of maximizing its effectiveness. The two groups will seek input from the scientific community, including investigators, scientific societies, academic institutions and health organizations, as well as from within NIH. The groups will study the context, criteria and culture of peer review to make sure the most talented individuals and reviewers are engaged in the process. More details, and the composition of the external and internal working groups, are available at the link at the top of this post.

Posted by Maxine Clarke on June 11, 2007
Categories: Systems | Permalink | Comments (0)

Reviewing appeals : Article : Nature Immunology

This month's Nature Immunology editorial (8, 541; 2007 -- link above); which is freely available, describes the journal's peer-review process and how it deals with appeals against decisions not to publish a submitted paper. This process is similar in all the Nature journals that publish original scientific research. From the editorial:

We believe that constructive critiques made during the review process improve manuscript quality, whether the manuscript is ultimately published in Nature Immunology or in another journal. Clarification of the review and appeal processes, as provided here, should assist authors in their preparation of manuscripts. We hope that by offering this advice to authors we can lessen frustration should a negative decision be rendered.

The full text of the editorial can be accessed via the link at the top of this post. Comments are welcome.

Posted by Maxine Clarke on May 25, 2007
Categories: Systems | Permalink | Comments (1)

Notes from the biomass: Warnock’s dilemma is a blog post on the discussion about lack of comments on scientific papers that are opened up for reader feedback online. Mr Warnock has a Wikipedia entry (which can be found via the Notes from the Biomass link) explaining why he thinks lack of comments does not (necessarily) indicate lack of interest.

Posted by Maxine Clarke on May 17, 2007
Categories: Systems | Permalink | Comments (0)

The Good, the Bad and the Ugly is the title of a post by Thomas Lemberger at the Seven Stones about how peer-reviewers might be rewarded. Nature is in the habit of thanking its thousands of peer-reviewers at the end of the year and providing a token of our appreciation, such as a special rate for subscription, but could we introduce other useful benefits? On occasion, for example, we have provided a letter of support for Green Card applications for regular peer-reviewers, and also have done the same for people's CVs.
But should peer-reviewers be penalised for being slow, or for providing a useless report? This is a tricky question, for reasons explained by Thomas at the Seven Stones, as well as others.

Posted by Maxine Clarke on April 27, 2007
Categories: Systems | Permalink | Comments (0)

In A Blog Around The Clock : Two Cultures, Coturnix writes about his surprise that the humanities seem more reluctant to experiment with peer-review systems than the scientific community. Coturnix describes his experiences of selecting posts for blog "carnivals" (themed collections of posts on one blog by various authors), and suggests that a peer-reviewed online blog-journal is the next logical step, or as he puts it: "I'd love to see publication of blogging anthologies collecting the best annual output by medical, environmental, education and humanities bloggers."

Paolo Massa has collected some links to articles with a similar theme.

Posted by Maxine Clarke on April 16, 2007
Categories: Systems, Technical solutions | Permalink | Comments (0)

Nature publishes a News story today (5 April) about China's plans to modernize traditional medicine. (Nature 446, 590-591; 2007; subscription or site licence required). The article describes China's plans to bring traditional medicine in line with modern standards, but reports scepticism among many about whether the research will reach "the scientific standards necessary for international recognition".

Some critics also worry that the plan doesn't set strict enough scientific standards. Although clinical research is listed as a priority, the plan doesn't specify whether there should be randomized, controlled trials in which neither practitioners nor patients know who is receiving active remedy and who is getting a placebo. And there is no requirement for TCM researchers to publish in internationally recognized journals. "Most research on TCM in the past is of poor quality, and is published only in Chinese medical journals without proper peer-review processes," remarks Wang [director of the National Centre for Drug Screening at the Shanghai Institute of Materia Medica]. "Without a clear position from the government, it is unlikely that the situation will change."

Other similar concerns, and the question of whether traditional medicine is even susceptible to a mechanistic approach, are discussed in the News article.

Posted by Maxine Clarke on April 05, 2007
Categories: Systems | Permalink | Comments (2)

The Valve - A Literary Organ | Idea for Discussion: An Academic Blog Review

Amardeep Singh, of The Valve literary review, proposes a system of academic blog reviewing, whereby people self-select individual blog posts they’ve written for review by others, via a community system such as proposed in some of the contributions to Nature's peer-review debate (see left-hand vertical column for a link). The proposal is intended to help those with academic blogs to be able to "publish", or at least, to "credentialize", selected shorter posts in addition to their more formal publications in the literature.

Posted by Maxine Clarke on March 29, 2007
Categories: Systems | Permalink | Comments (0)

Open Call From the Patent Office - washingtonpost.com

The Washingtonpost.com reports (link above) that the US Patent and Trademark Office is starting a pilot project that will not only post patent applications on the Web and invite comments, but also use a community rating system designed to push the most respected comments to the top of the file, for serious consideration by the agency's examiners. The Washingtonpost.com article compares the the system to that used by Wikipedia, the popular user-created online encyclopedia.

The "peer to patent" project starts next week (week of 2 April) with a pilot programme, led by the Institute for Information Law and Policy at New York Law School and the US Patent and Trademark Office. The pilot will initially accept 250 patent applications from companies including IBM, Intel, Oracle, Hewlett-Packard and Microsoft.

Posted by Maxine Clarke on March 28, 2007
Categories: Policy and Public, Systems | Permalink | Comments (0)

According to an article in Wired news on 14 March, the "$10 billion science publishing industry hasn't heard the last of a bill that would make publicly funded studies available for free. Sen. John Cornyn (R-Texas) has pledged this year to resurrect the Federal Research Public Access Act (S.2695), which would require federally funded research to become publicly available online within six months of being published."

The Wired article refers to a news story in Nature of 24 January, which describes how a group of journal pubishers have employed a public relations firm to "take on" the 'open-access' movement, to help communicate that the issue is not only about preserving profits, as some of the open-access proponents claim. From Wired :

"Our core message is that we believe in the integrity of the peer-review system and the investments in it," said Brian Crawford, chairman of the executive council of the Professional and Scholarly Publishing Division of the Association of American Publishers. "It's inappropriate for the government (to interfere)."

Later in the article: "A few traditional subscription publishers are also experimenting with open access, making more research freely available after a certain amount of time or if authors pay extra fees. Nature, for example, is experimenting with a free journal and allowing some authors to pay to make their findings publicly available. And earlier this month, the Howard Hughes Medical Institute announced it would pay the large Elsevier publishing house to make institute-funded research available to the public for free after six months. The change will take effect on Sept. 1, affecting research published after that date.

"But it is already clear that a journal like Nature would struggle under an open-access business model," said David Hoole, head of brand marketing for the Nature Publishing Group. "We reject 90 percent of the articles we receive, and spreading the cost of peer review over the few authors who do get published would be very unfair (and would probably deter submissions). We [Nature] have approximately 1,000 authors, and 60,000 subscribers [a year]. It seems fairer to spread the costs over the subscribers." "

Posted by Maxine Clarke on March 16, 2007
Categories: Systems | Permalink | Comments (2)

Richard Akerman, in a 2 March post on Science Library Pad entitled "Soft peer review", provides links to some interesting new reports on the subject. One is called "Social software and new opportunities for peer review", via Library 2.0; another is "Soft peer review? Social software and distributed scientific evaluation" via Academic Productivity blog.
Richard is one of the contributors to Nature's web focus on peer review, published last year and archived in full on Peer to Peer (it contains 22 articles, including Richard's, and remains open to comments from the community of peer-reviewers, authors and scientists).
In his Science Library Pad post, Richard concludes: "I do think the debate about "open" peer review vs. traditional peer review is a bit of a red herring, and it very much concerns me when people suggest that open review can replace traditional (or in the language of this posting "hard") peer review. We have already had open peer review for years, it's called preprint feedback, mailing lists, ArXiV, letters to the editor... it's a tremendous addition to, but not replacement for, the rigourous anonymous peer review system needed to provide a publication filter."

Posted by Maxine Clarke on March 06, 2007
Categories: Systems | Permalink | Comments (0)

Dennis McDonald - Managing Technology - A Comparison of Blogging and Journal Peer Review
More comparisons between journal peer-review and blogging.
In these two posts on his blog All Kind Food, technology consultant Dennis McDonald attempts to encapsulate the key features that distinguish peer-reviewed journals from blogs. It is an ambitious challenge, to compare the formal structure of journal publication with the "anything goes" nature of the blogosphere, where last week might as well be last century.
Nevertheless, Mr McDonald has identified some key similarities: the motivation of prestige and reputation (journal impact measures or blog rankings in indexes like Technorati); "secondary publishing" via abstracting and indexing (journals) or the equivalent for blogs (Connotea, Delicious, Technorati listings); citations (formal references for journals or links and trackbacks for blogs), and various types of social network.
Although the publishing processes may be similar, blogs are not peer-reviewed. While acknowledging this crucial difference, Mr McDonald believes that it is these social networking aspects of both media where the peer-review process could integrate with the blogging system:
"The information contained in a refereed and published journal article, for example, may have been previously disseminated to other researchers via email, workshops, conferences or conference proceedings, or final reports submitted to funding agencies. In other words, the information that appears in a refereed journal article may have long been communicated to members of the author's existing social and professional networks, especially to those working in the same or in very closely related technical or professional areas. It's not unusual, for example, that research reported in a journal article has been long since superseded by other work done by the author.
The article itself, while it now becomes available to a much wider audience through the a wide range of physical and electronic access channels, acts not only as a conduit of research information but also an advertisement of the skill and accomplishments of the author, filtered by the "halo effect" of prestige and recognition of the journal in which it is published."

Posted by Maxine Clarke on January 30, 2007
Categories: Systems | Permalink | Comments (0)

Cognitive Daily: Nature's report on open peer review

I'm a bit behind on writing posts because of the launch of the author and referee's website earlier this week (www.nature.com/authors) Somewhat belatedly, therefore, but none the worse for it, I'm drawing attention to a post on Cognitive Daily (link at top of this post) about Nature's peer-review trial, and about the odds for open peer review in general.

In his post, Dave Munger writes: "What kind of incentives would work? The most obvious would be career incentives: if work as a reviewer was rewarded with tenure and promotion, it would soon become one of the top priorities of any scholar. Unfortunately this revolutionary change in the glacial world of academia is about as likely as PZ Myers undergoing a religious conversion, so we probably will need to look elsewhere. Many journals already require authors to review articles as a condition of submitting articles for publication. Perhaps this sort of incentive could be adapted to an open review process. Even so, it would be difficult to administer. How would reviewers be evaluated? By authors? But then wouldn't there be an incentive for reviewers to rubber-stamp articles for publication? For now, it appears that the peer review process as it stands might be the lesser evil."

Dave concludes that some combination of wikis and blogs might one day partially replace traditional peer reivew. But, he says, "Only when contributing to these resources becomes part of the tenure rewards system are they likely to become important factors in the world of academic publishing."

Posted by Maxine Clarke on January 13, 2007
Categories: Systems | Permalink | Comments (0)

Theodora Bloom

The Internet is allowing much more interactive science publishing

Online tools can be used to improve the accuracy, transparency and usefulness of the scientific literature by moving away from the traditional emphasis on closed peer review. Given the capability for post-publication amendment of articles, the scientific articles themselves and the peer-review process will soon be profoundly different from today’s standard.

Continue reading "Systems: Online frontiers of the peer-reviewed literature" »

Posted by Maxine Clarke on June 29, 2006
Categories: Peer review debate, Systems | Permalink | Comments (2)

Erik Sandewall

A hybrid system of peer review

Traditional peer review serves two purposes: to give feedback to the authors, helping them to improve their manuscript; and to control the quality of published articles. I believe that more value can be obtained by incorporating an open element in the peer-review system.

Continue reading "Systems: Opening up the process" »

Posted by Maxine Clarke on June 20, 2006
Categories: Peer review debate, Systems | Permalink | Comments (0)

Brenda Riley

Database publication presents unique challenges for the peer reviewer

The reader of a scientific paper in a high-quality journal knows that the information has been vetted by a formal process of peer review, moderated by editors. But the face of publishing is changing, and peer review of databases is becoming an increasingly important facet of scientific data curation. The Signaling Gateway's molecule pages represent an important, innovative experiment in applying models of peer review developed in journals to the much newer world of scientific databases.

Continue reading "Systems: Trusting data's quality" »

Posted by Maxine Clarke on June 20, 2006
Categories: Peer review debate, Systems | Permalink | Comments (0)

Thomas Koop and Ulrich Pöschl

The editors of Atmospheric Chemistry and Physics explain their journal’s approach.

Recent high-profile cases of scientific fraud have fuelled the discussion of scientific quality control. A problem of similar, if not greater, importance is the large proportion of carelessly prepared scientific papers that dilute rather than enhance scientific knowledge. Both problems indicate shortcomings in the traditional peer-review system. Many scientists and publishers believe that peer review remains the best available approach for quality assurance, but requests for improvements are commonplace.

Continue reading "Systems: An open, two-stage peer-review journal" »

Posted by Maxine Clarke on June 14, 2006
Categories: Peer review debate, Systems | Permalink | Comments (2)

Eugene Koonin, Laura Landweber, David Lipman and Ros Dignon

Can 'open peer review' work for biologists? Biology Direct is hopeful.

The advent of immensely powerful means of communication in our information age offers unprecedented opportunities for experimentation with new approaches to scientific publishing. In an attempt to offer the scientific community an alternative to the current peer-review system, we recently launched a new journal, Biology Direct.

Continue reading "Systems: Reviving a culture of scientific debate" »

Posted by Maxine Clarke on June 05, 2006
Categories: Peer review debate, Systems | Permalink | Comments (1) | TrackBacks (1)