Peer-to-peer

An article in the Financial Times, Science stifled? Why peer review is under pressure (11 June 2008), reports various recent criticisms of the peer-review system, including a letter to the newspaper by 25 distinguished scientists calling for a "global fund to support inspired scientists, free of peer review"; news of a Royal Society pilot scheme for a “blue skies” research fund, to avoid the "constraints of conventional peer review by using a generalist panel to consider proposals from any field, on the basis of their novelty and potential to open up new areas of science and technology"; and in the announcement of this year's Grand Challenges programme of the Bill and Melinda Gates Foundation, Tachi Yamada, the foundation's head of global health, is cited as saying "We've got to get around peer review – it’s anathema to innovation. Innovation has no peers, by definition.”
The Financial Times article goes on to identify various innovations in the peer-review process itself, being tried or in normal use by various publications. Scientists themselves, however, choose to publish in the highest quality journals rather than on the basis of their peer-review systems. Linda Miller, US Executive Editor of Nature, is quoted in the article:

Linda Miller, executive editor of Nature, agrees that scientists continue to seek publication in prestigious journals to enhance their own standing. They also concentrate on reading the best-regarded ones, precisely because their time is precious. “You want to be directed, to use the best journals as a filtering device,” she says. “I have been an editor for more than 20 years and I have handled a lot of papers. Every single one has been improved by peer review.”

Posted by Maxine Clarke on June 25, 2008
Categories: Perspectives | Permalink | Comments (3)

Today, 18 June, is the first anniversary of Nature Precedings, where researchers can post their unpublished manuscripts, presentations, posters, white papers, technical papers, supplementary findings and other scientific documents, which can all be "peer-reviewed" online by anyone in the scientific community. (The website was available before June 2007 in 'beta' form.) Santosh Patnaik, a user who periodically tracks Nature Precedings at the Nature Network Nature Precedings forum, estimates that the 500th document will be uploaded some time in the next two weeks.
Because the code for Nature Precedings is freely available, Dr Patnaik has mined some data to chart the growth of the website. His results are presented here, in graphical form. The number of posters and presentations, common when the site first launched, is now barely increasing, whereas the number of manuscript uploads has grown at a steady rate over the past year. The most popular discipline, perhaps unsurprisingly, is bioinformatics, although most other disciplines are also becoming more popular, particularly neurosience, 'evolution and ecology', and chemistry. (For those interested in statistics, Dr Patnaik has also estimated the productivity of Nature Precedings authors.)
One aspect of this type of open peer-review is that discussion is not limited to the English language, even though the language of uploaded documents is in English. The vast majority of comments are, however, in English: here is an example of constructive review in the neuroscience field, Nature Precedings style. There are many other examples: the most active discussions are here, but one can also filter by subject area.

Update: Hilary Spencer and the Nature Precedings team provide a one-year perspective at Nature Network.

Posted by Maxine Clarke on June 18, 2008
Categories: Technical solutions | Permalink | Comments (0)

A News story in the 12 June issue of Nature (453, 823; 2008) by Meredith Wadman:
Responding to hundreds of critical comments, the US National Institutes of Health (NIH) has reversed several controversial proposals made in February as part of a year-long effort to overhaul the agency's peer-review system (see Nature 451, 1035; 2008).
As part of an initiative called Enhancing Peer Review, announced in a finalized form on 6 June, the agency will spend at least $200 million annually over the next five years to foster groundbreaking, investigator-initiated research. Of that, at least $250 million will go to a new beast: a Transformative R01 Award, a reach-for-the-skies version of the NIH's basic grant. The remaining $750 million will go to existing awards that reward risk and innovation: the Eureka, New Innovator and Pioneer awards.
The changes “are concrete solutions that will maximize flexibility, remove any unnecessary burden, stimulate new innovation and promote transformative research”, says NIH director Elias Zerhouni.
They include rewards for long-serving reviewers; a streamlined, 12-page R01 grant application, down from 25; and a seven-point, integer scoring scale for grant applications, which will be assessed across five criteria: impact, investigators, innovation, feasibility and environment. Current applications are graded on a 41 point scale, from 1.0 to 5.0, raising complaints that they claim a degree of accuracy that can't be scientifically defended.
Among the controversial proposals shelved by the agency was a recommendation that all applications, even those on a second or third submission, would be treated as new, without reviewer access to prior reviews.
Gone, too, is the category “not recommended for resubmission”, which had been suggested for dismal applications. Scientists felt that branding projects with “a clear, checkbox-driven stigma is bad, that it could have unintended consequences”, Jeremy Berg, director of the National Institute of General Medical Sciences, told the advisory committee.
Berg and Lawrence Tabak, director of the NIH's dental institute, head the group that developed the recommendations and are charged with implementing them over the next 18 months.
The agency also jettisoned a “minimum effort requirement” that would have required principal investigators to commit at least 20% of their time to any single NIH grant — an item of particular concern for 'grandee grantees' (see Nature 452, 258–259; 2008). Instead, grantees will need to indicate if they will have more than $1 million in cumulative NIH funding.

Posted by Maxine Clarke on June 12, 2008
Categories: Grants, tenure and positions, Systems | Permalink | Comments (2)

Stephen T Huang, Maged N Kamel Boulos and Robert P Dellavalle write an article in the June issue of EMBO Reports (9, 496-499; 2008) with the title: Scientific discourse 2.0. Will your next poster session be in Second Life?
From the article:

Certainly, peer-reviewed literature and scientific meetings in the physical world will remain the main modes of distributing scientific information and informal communication. Yet, communication through virtual-world technology might become a useful supplement to the traditional discourse. The particular strengths of this technology include: its potential to share, review and comment on information, both with the public and one's peers; options that allow users to create and develop unique objects, and presentations to educate and inform others and to display data; and, last but not least, the time and cost of bringing people together within and across disciplines can be reduced.
As with any new technology, there are issues that could have an impact on the usefulness of online communication and its acceptance within the scientific community. Scientists who rely on peer-reviewed data for their work might find Web 2.0's lack of proofreading unacceptable to document research findings. However, we should explore the existing and potential applications of virtual communication for unique ways to discuss ideas, answer questions, educate and debate. Our ability to understand what we can accomplish in online worlds depends on our collective experience with the technology. The more scientists and clinicians who work with and comprehend the applications of virtual worlds for their respective research fields, the sooner we will realize how this technology can be best applied. The next step is to invite ourselves into these online realms, experiment with what they have to offer, and see where our exploration and creativity takes us.

(Stephen T. Huang is at Northwestern University's Feinberg School of Medicine in Chicago, Illinois, USA; Maged N. Kamel Boulos is at the University of Plymouth, UK; Robert P. Dellavalle is at the Denver Veteran's Affairs Medical Center and the University of Colorado Denver, Denver, Colorado, USA.)

Posted by Maxine Clarke on June 10, 2008
Categories: Technical solutions | Permalink | Comments (0)

The Editorial 'Working double-blind' (Nature 451, 605–606; 2008), also republished on this blog and stimulating more than 70 comments, referred to a study (1) that found more female first-author papers were published using a double-blind, rather than a single-blind, peer-review system. The data reported in ref. 1 have now been re-examined (2). The conclusion of ref. 1, that Behavioral Ecology published more papers with female first authors after switching to a double-blind peer-review system, is not in dispute. However, ref. 2 reports that other similar ecology journals that have single-blind peer-review systems also increased in female first-author papers over the same time period. After re-examining the analyses, Nature has concluded that ref. 1 can no longer be said to offer compelling evidence of a role for gender bias in single-blind peer review. In addition, upon closer examination of the papers listed in PubMed on gender bias and peer review, we cannot find other strong studies that support this claim. Thus, we no longer stand by the statement in the fourth paragraph of the Editorial, that double-blind peer review reduces bias against authors with female first names.
References
1. Budden, A. E. et al . Trends Ecol. Evol. 23, 4–6 (2008).
2. Webb, T. J. , O'Hara, B. & Freckleton, R. P. Trends Ecol. Evol. doi:10.1016/j.tree.2008.03.003 (2008).

Posted by Maxine Clarke on June 05, 2008
Categories: Ethics, Quality and value, Systems | Permalink | Comments (2)

In the latest Nature Reports Stem Cells Inside the Paper feature, senior Nature editor Natalie DeWitt discusses the paper by H. H. Chang et al. 'Transcriptome-wide noise controls lineage choice in mammalian progenitor cells', published in Nature 453, 544-547 (2008).

The Editor's summary of this paper: Even in clonal populations of cells, there is significant phenotypic variation from cell to cell. This could reflect the 'noise' inherent in gene expression: or the various cell states could represent stable phenotypic variants. Chang et al. analysed the behaviour of an 'outlier' in clonal populations of mouse haematoipoietic stem cells that had very high expressions of the stem cell marker Sca-1 and found that outliers possessed distinct transcriptomes. Though the transcriptomes eventually reverted back to that of the median cells, while they differed they could drive the cells to express characteristics of distinct cell fates. Thus clonal heterogeneity of gene expression may not be due to noise in the expression of individual genes, but rather is a manifestation of metastable states of a slowly fluctuating transcriptome. These fluctuations may govern the reversible, stochastic priming of multipotent progenitor cells in cell fate decision.

The Nature Reports Stem Cells article provides the details of what the peer-reviewers thought of this paper when it was submitted, and how the authors responded. The initial view of the three peer-reviewers can be summarized as follows:

Reviewer 1 In summary, we believe that this paper reports a novel and important biological mechanism for differentiation: it gives evidence that slow stochastic variations in stem cell state last for a long time and result in different fates. It relates the recent finding of slow fluctuations in human protein levels to the biological outcome of cell fate, and finds long lasting differences in transcriptomes in different subpopulations. It is an excellent choice for Nature, provided that the comments are addressed.

Reviewer 2 Although the phenomenon described is immensely interesting and the idea of heterogeneity being retained within even clonal populations of cells is plausible, the authors merely describe this phenomenon and in some instances, do not provide conclusive data to support their interpretation. If a mechanism was determined this would definitely aid in its novelty and interest. In its current form, I believe this manuscript should not be accepted at Nature.

Reviewer 3 This is a very timely and interesting paper, that should be of interest to a broad range of researchers. However, I have some serious concerns over the modeling.

The details of the reviewers' concerns, and the authors' response, can be read at Nature Reports Stem Cells,

You can also read a set of question-and-answer sessions between editor Monya Baker and several of the peer-reviewers of the recent Nature Insight on Regenerative Medicine, here on The Niche, the blog of Nature Reports Stem Cells.

Posted by Maxine Clarke on June 02, 2008
Categories: Systems | Permalink | Comments (0)