Peer-to-peer

From Nature News in Brief (454, 564; 2008):
The system used by the US National Institutes of Health (NIH) to evaluate grant proposals does not adequately compensate for reviewer bias, affecting one in four proposals, a study finds.
Valen Johnson, a biostatistician at the University of Texas M.D. Anderson Cancer Center in Houston, evaluated reviews for nearly 19,000 grant proposals performed by around 14,000 reviewers in 2005 (V. E. Johnson Proc. Natl Acad. Sci. USA doi:10.1073/pnas.0804538105; 2008).
Each application is typically read by 2–5 reviewers, then discussed when a larger study section of about 30 reviewers meets. In the end, scores from all the study section's members — readers and nonreaders alike — are averaged together.
The system fails to account for individual bias and places undue weight on panel members who have not read the proposals, Johnson argues. He found that the top grants were largely unaffected by reader bias, but that it did affect grants closer to the funding cut-off line. Overall, accounting for reader bias changed about 25% of the funding decisions, meaning that one in four funded proposals would have been replaced by one that had not been funded.

See here for a longer version of this story, which includes an informative online readers' discussion.

Posted by Maxine Clarke on August 14, 2008
Categories: Ethics, Grants, tenure and positions | Permalink | Comments (0)

From the Editorial in the August issue of Nature Reviews Cell and Molecular Biology (9, 583; 2008):

First-time applicants [for NIH grants] compete with thousands of new and established scientists, an experience that can be intimidating and frustrating. The level of detail required for the majority of applications — for example, the exhaustive budgetary specifications — has been a burden not only for applicants, but also for reviewers. Moreover, a grant proposal can take up to 18 months to pass through the system, waiting in line behind older applications, most of which must go back and forth to applicants for amendments before approval. Of course, funding is not guaranteed: the NIH receives between 35,000 and 40,000 proposals a year, of which only 25–30% will eventually be funded.
On the basis of feedback solicited from the life-sciences community on the current peer-review system, the NIH plans to revamp the grant-review process to encourage innovative research and reward quality science. The new practice involves shortened applications that should improve the value and transparency of the review process and ensure balanced, fair reviewing across scientific fields and scientific career stages. Recruiting, training and rewarding the best reviewers is also a priority. The new process allows reviewers to focus more on the science presented and less on the details of financial requests, and compensates them for their time and effort.
Hopefully, the NIH's changes will not only foster new innovative research efforts, but they will also free up some of scientists' precious time — allowing them to spend more time planning experiments, on the bench and writing and reading scientific papers.
See also: Will NIH's overhaul be cosmetic or curative?

Posted by Maxine Clarke on August 06, 2008
Categories: Grants, tenure and positions | Permalink | Comments (0)

A News story in the 12 June issue of Nature (453, 823; 2008) by Meredith Wadman:
Responding to hundreds of critical comments, the US National Institutes of Health (NIH) has reversed several controversial proposals made in February as part of a year-long effort to overhaul the agency's peer-review system (see Nature 451, 1035; 2008).
As part of an initiative called Enhancing Peer Review, announced in a finalized form on 6 June, the agency will spend at least $200 million annually over the next five years to foster groundbreaking, investigator-initiated research. Of that, at least $250 million will go to a new beast: a Transformative R01 Award, a reach-for-the-skies version of the NIH's basic grant. The remaining $750 million will go to existing awards that reward risk and innovation: the Eureka, New Innovator and Pioneer awards.
The changes “are concrete solutions that will maximize flexibility, remove any unnecessary burden, stimulate new innovation and promote transformative research”, says NIH director Elias Zerhouni.
They include rewards for long-serving reviewers; a streamlined, 12-page R01 grant application, down from 25; and a seven-point, integer scoring scale for grant applications, which will be assessed across five criteria: impact, investigators, innovation, feasibility and environment. Current applications are graded on a 41 point scale, from 1.0 to 5.0, raising complaints that they claim a degree of accuracy that can't be scientifically defended.
Among the controversial proposals shelved by the agency was a recommendation that all applications, even those on a second or third submission, would be treated as new, without reviewer access to prior reviews.
Gone, too, is the category “not recommended for resubmission”, which had been suggested for dismal applications. Scientists felt that branding projects with “a clear, checkbox-driven stigma is bad, that it could have unintended consequences”, Jeremy Berg, director of the National Institute of General Medical Sciences, told the advisory committee.
Berg and Lawrence Tabak, director of the NIH's dental institute, head the group that developed the recommendations and are charged with implementing them over the next 18 months.
The agency also jettisoned a “minimum effort requirement” that would have required principal investigators to commit at least 20% of their time to any single NIH grant — an item of particular concern for 'grandee grantees' (see Nature 452, 258–259; 2008). Instead, grantees will need to indicate if they will have more than $1 million in cumulative NIH funding.

Posted by Maxine Clarke on June 12, 2008
Categories: Grants, tenure and positions, Systems | Permalink | Comments (2)