From Nature News (Nature 460, 676; 2009), by Meredith Wadman:

Applicants for the coveted Challenge Grants issued by the US National Institutes of Health (NIH) as part of the American Recovery and Reinvestment Act learned the peer-review scores for their proposals late last month. Yet they received little in the way of certainty over whether those scores will translate into money come September, when the NIH will announce which grants it plans to fund.

Competition for the US$1-million, two-year awards is fierce — the agency in Bethesda, Maryland, received more than 21,000 applications, and the NIH director’s office will fund only about 1% of these. With ordinary grants, applicants can usually tell if their grant is fundable as soon they receive their percentile score because they already know the designated ‘payline’, or percentage of fundable applications. The NIH has designated an initial $200 million of $10.4 billion in economic stimulus funds for the grants, but with so many variables at play in allocating the stimulus money, predicting whether a given score will land funding is almost impossible — meaning that those with percentile scores in the mid-single digits are left hanging.

Meanwhile, the burden on the thousands of grant reviewers has, according to some, turned out to be bearable. Gary Johnson, chairman of the pharmacology department at the University of North Carolina, Chapel Hill, told the NIH that he could review up to five Challenge Grant applications. “And they only gave me a couple,” he says. “I don’t know anyone who was overwhelmed by reviewing, because there was an overwhelming agreement of investigators to participate in the process.”

NatureJobs reports on the contentious topic of possible gender bias in peer review (Nature 459, 602; 2009). Peer review assesses what is of value in science, yet it has been criticized for biases. One such perceived bias is gender, although evidence for such a bias has been contradictory. A 2007 meta-analysis (L. Bornmann et al. J. Informet. 1, 226–238; 2007; reported at the time in NatureJobs Nature 445, 566; 2007) concluded that women are at a disadvantage in peer review of funding applications. As this study incorporated all known research on this issue, it was suggested to be definitive.

Nevertheless, H. W. Marsh et al. (Am. Psychol. 63, 160–168; 2008) presented conflicting results the following year. This study was comprehensive, based on data from the Australian Research Council (10,023 reviews by 6,233 external assessors of 2,331 proposals from all disciplines), and concluded that the gender of the applicant had no effect on the outcomes of peer review, irrespective of the discipline, gender and nationality of the reviewers, and whether reviewers were selected by a funding panel or chosen by the applicants.

Why should these two studies have conflicting results? To investigate, Marsh and Bornmann now report in NatureJobs how both research teams worked together to reanalyse the data and extend the original meta-analysis. They describe how they applied new, stronger statistical approaches to 66 sets of results representing 353,725 proposals from 8 countries. In this extended study, which will be published in Review of Educational Research, they found no effect of the applicant’s gender on the peer review of their grant proposals. This lack of effect held across country, year of publication of the studies included in the meta-analysis, and disciplines ranging from physical sciences to the humanities.

At least for grant applications, all of the co-authors from each of the research teams agree that the weight of evidence suggests that the applicant’s gender has no effect on the outcome of peer review, and that these findings are robust and broadly generalizable.

Herbert Marsh is a professor of education at the University of Oxford, UK

Lutz Bornmann is a PhD student at the ETH University in Zurich, Switzerland.

This Peer-to-Peer post is an edited version of their NatureJobs article (which is free to access online).

There is some discussion of this NatureJobs article and the timing of its publication at Nautilus blog.

Note: readers interested in the Review of Educational Research paper in advance of its publication can obtain a copy directly from Dr Bornmann or from the NatureJobs editors.

After a campaign by scientists, the UK Engineering and Physical Sciences Research Council (EPSRC) has softened and delayed its controversial policy to bar serially unsuccessful grant applicants from making funding bids for one year (Nature online News, 5 May 2009). The ban was due to be imposed on 229 researchers starting on 1 June, in an effort to reduce pressure on an overloaded system that currently peer-reviews all grant applications. But eight weeks after it published the policy (see Nature online news 19 March 2009 and Nature 458, 391; 2009), the EPSRC now says that the restriction will not come in until 1 April 2010 — giving scientists more time to change their grant-submission behaviour so that they do not fall under criteria defining repeated failure. And instead of being excluded outright, researchers will be allowed one application during the year.

“We have made these adjustments to address concerns raised by the community — for example, the retrospective nature of [the policy’s] implementation,” the EPSRC said in a statement. “We’ve made bold changes to protect peer review, but we’re not an insensitive organization.”

Peter Main, director of education and science at the Institute of Physics in London, says the EPSRC has listened to criticism and has shown flexibility. “It’s the policy that it perhaps should have been in the first place,” says Joe Sweeney, an organic chemist at the University of Reading, UK, who set up an online petition demanding the policy be repealed, signed by more than 1,900 scientists.

But some researchers say they are disappointed not to have been consulted more directly beforehand — which might have prevented the EPSRC from introducing the ban in the first place. “It’s something of a shame that we had to force them into this policy change,” says Philip Moriarty, a physicist at the University of Nottingham, UK.

The EPSRC is keeping a policy introduced on 1 April, to refuse uninvited resubmissions of failed proposals, which it says will cut 20% of applications submitted for review. The exclusion policy had been expected to cut a further 10%.

The EPSRC says that letters intended to warn individuals in April were never sent. “We are an organization that listens to the community,” says chief executive David Delpy. “If we can make amendments to help researchers whilst ensuring the overall policy is still effective, then that’s in everyone’s interest.”

Not all scientists oppose the proposals, however. In an online comment to this Nature news story, Nial Wheate writes: “I don’t have a problem with the EPSRC policy. Seriously, if you have three grants all ranked in the bottom half in one year, then you need to stop and think about what research you are trying to get funding for. And what’s wrong with only submitting 2 grants per year to make sure this doesn’t happen to you? The policy makes sense to me.”

News story first published online 5 May 2009 Nature doi:10.1038/459020b.

Frank Gannon imagines a modern response to Darwin’s research grant application (EMBO Reports 10, 1; 2009):

“We were a bit puzzled by your handwritten application for funding as it shows a staggering disregard for our practices and rules relating to requests for research support. We usually receive applications by electronic submission, and they must be received before 5 pm on the closing date. However, we realize that our current instructions do not formally preclude non-electronic communication and therefore we had little choice but to consider your application.

We sent your proposal to a panel of international expert referees who have not worked or published with you during the past 10 years. Their comments are summarized below and we hope that you will find them helpful should you choose to resubmit in the future. Although your project departs from the routine—a refreshing change from modern research projects that tend to be merely incremental steps with no real impact on the universe of relevant knowledge—I am afraid that your application for funding has been rejected.”

Read on for the details of the three reviewers’ reports, and why the potential funder concludes: “There are unusual and therefore intriguing aspects to your idea and the scale of your ambition is impressive. In the final analysis, however, our decision is based on a simple test of whether the work will have a major scientific impact—in other words, will people consider it to be an important piece of knowledge in a hundred years’ time? Regretfully, we do not think that your proposed work will yield sufficient insight to meet this requirement.”

Nature‘s special set of articles to celebrate Darwin’s achievements is available here.

Via Branwen Hide at Nature Network, the Biotechnology and Biological Sciences Research Council (BBSRC) has today announced changes to the way its peer-review committees are organized, the way new research and policy priorities are highlighted and a reorganization of funding structures for its five sponsored Institutes. The research council website summarizes the changes:

-The creation of four new research committees from the existing seven

-The setting up of a mixed economy of peer review membership, including core committee members and a pool of reviewers able to be called on flexibly for their specific expertise

-New research and policy priorities that will overarch all of BBSRC’s activities. The policy priorities will help BBSRC researchers to consider the strategic relevance of their proposals when they make applications

-A system of highlight notices that BBSRC will use to generate demand when it identifies the need for more applications in certain areas

-Institute Strategic Programme Grants to replace Core Strategic Grants to the BBSRC-sponsored Institutes

The BBSRC is holding a series of roadshows during November to explain and discuss the changes. Presentations will be available to download from 15 November (2008).

Another UK research council, the Engineering and Physical Sciences research council, is also currently undertaking a review of its peer-review processes.

The UK Engineering and Physical Sciences Research Council is currently running a survey on peer-review, from 9 October to 7 November 2008. The survey aims to gather opinions about the peer-review process to help the research council to make improvements in its processes and procedures. From the survey outline: “We want to gather as much information and opinion about the current EPSRC peer-review process as possible so that we can make it more effective and fit for purpose to meet future needs, and improve its usability for applicants and reviewers while ensuring it remains fair and transparent.” Anyone with an interest in EPSRC can respond, although the survey’s questions are aimed at scientists who have either reviewed for or applied for grants from the council.

Further information about the EPSRC’s reviewing process can be found here, including standards, advice on peer-reviewing and how to respond to peer-reviewers’ comments, and details of the council’s incentive scheme for reviewers.

From the Editorial in the August issue of Nature Reviews Cell and Molecular Biology (9, 583; 2008):

First-time applicants [for NIH grants] compete with thousands of new and established scientists, an experience that can be intimidating and frustrating. The level of detail required for the majority of applications — for example, the exhaustive budgetary specifications — has been a burden not only for applicants, but also for reviewers. Moreover, a grant proposal can take up to 18 months to pass through the system, waiting in line behind older applications, most of which must go back and forth to applicants for amendments before approval. Of course, funding is not guaranteed: the NIH receives between 35,000 and 40,000 proposals a year, of which only 25–30% will eventually be funded.

On the basis of feedback solicited from the life-sciences community on the current peer-review system, the NIH plans to revamp the grant-review process to encourage innovative research and reward quality science. The new practice involves shortened applications that should improve the value and transparency of the review process and ensure balanced, fair reviewing across scientific fields and scientific career stages. Recruiting, training and rewarding the best reviewers is also a priority. The new process allows reviewers to focus more on the science presented and less on the details of financial requests, and compensates them for their time and effort.

Hopefully, the NIH’s changes will not only foster new innovative research efforts, but they will also free up some of scientists’ precious time — allowing them to spend more time planning experiments, on the bench and writing and reading scientific papers.

See also: Will NIH’s overhaul be cosmetic or curative?