News blog

NIH sticks with ‘two strikes’ grant rule – updated

The US National Institutes of Health (NIH) will continue a policy that allows grant applicants just one resubmission if their proposal is rejected the first time, the agency announced today.

The policy, launched  in 2009 as part of an overhaul of peer review at the NIH, “continues to achieve the stated goals of enabling NIH to fund as much meritorious science as possible in as short a time period as possible,” Sally Rockey, the NIH deputy director for extramural research, wrote on her blog Rock Talk.

The ‘two-strikes’ policy has received negative reviews from the grantee community, many of whom argue that its effect, in an era of historically low grant-funding levels, is to randomly cull proposals of high merit.

The decision “is a characteristic refusal by administrators to admit they erred badly when they made a change that has been consistently unpopular with those it affected, in this case NIH grant applicants,” says John Moore, a senior HIV/AIDS scientist at Weill Cornell Medical College in New York, and a longtime NIH grantee.

Rockey based her decision on new data the agency has generated that continue through the end of the government’s 2011 fiscal year.  As shown in Figure 2 (below) from Rockey’s blog, the average time that scientists wait to receive an award has decreased from more than 80 weeks to 56 weeks since the policy was implemented.

NIH

As we reported last month in ‘NIH re-evaluating “two strikes” rule’:

“The NIH’s rationale for the 2009 change was that the [chance to submit an application a third time] was causing peer reviewers, either consciously or subconsciously, to favour second and third submissions over first-time proposals, creating, in effect, a queue similar to airplanes circling an airport waiting for a free runway to allow them to land.”

Today, Rockey wrote, “the ‘queuing’ of original applications and revisions has been eliminated.” In addition, she noted, a higher proportion of first-time applications are being funded compared to second-time attempts under the new rules, an outcome that was not necessarily expected. In 2011, first-time applications represented 49% of those funded, and second-time applications, 44%. (The remaining 7% were residual third attempts, left over from before the new policy was instituted.)  Rockey’s Figure 1 illustrates this:

NIH

The agency also used new data to argue that its policy is not disproportionately hurting the youngest investigators, who have less experience crafting grant applications — and fewer substantially different experiments to fall back on in designing a new application should they ‘strike out’ after two attempts. Figure 2 also shows that, under the two-strikes regime, new principal investigators waited only a couple of weeks longer to win an award than did all applicants collectively.

Rockey also addressed a suggestion that second-submission applications scoring in the top 25% be allowed a third try. Her modelling (Figure 3) showed that, if all the failed second-chance applications were funded on a third try, the agency would have been able to fund 21% fewer first-time applications and 19% fewer second attempts.

NIH

The two-strikes policy has been the bane of many NIH-funded scientists, more than 2,300 of whom signed this 2011 letter to the NIH, pleading with the agency to return to a three-strikes system allowing a second resubmission of a failed proposal.

Rockey added that, despite holding fast to the changed policy, the agency isn’t oblivious to the pain being caused by stagnant budgets and historically low grant application success rates: “We are well aware that a lot of great science goes unfunded because paylines have decreased.”

UPDATE:  Robert Benezra, the cancer biologist at Memorial Sloan-Kettering Cancer Center in New York who spearheaded the 2011 letter of protest to the NIH, had the following reaction to today’s decision:

“The central premise of our argument was that study sections cannot distinguish between an unfunded [second-try application] at the 20th percentile versus a funded [first time submission] at the 12th percentile. I have not spoken with a single study section member who feels otherwise. Therefore that [second-try] applicant should be allowed to reapply as many times as he/she likes. Otherwise the system is being culled randomly in that quality range and it would be just as fair to have a lottery.

Yes, the new system decreases the time to funding, but it does so by a random selection of meritorious applications. This is all due to the fact that in the halcyon days of doubling NIH budgets, too many people were brought into the system. The NIH should just say this and we can all be on our way.”

Comments

  1. Report this comment

    Michael Andresen said:

    Sadly what is forgotten here is that peer review at its best is a deliberative and iterative process. This is true for scientific publications – good peer review actually improves, polishes and refines the ideas toward better quality. The two strikes rule really has reduced and degraded peer review at NIH. There are other factors such as the decline of stable study sections and experienced Scientific Research Officers. The idea that “wait in line” is dead really does not agree with experience or conversations with study section members or even human nature. There simply is not enough money. Politics and trendy ideas are displacing solid science. It is a common experience now that peer review at the best journals provides more appropriate critiques than evaluations from often rashly constructed review panels at NIH. Peer review at NIH is broken and the hierarchy defensively shops for hollow statistics and PowerPoint rationales that resemble the old IBM advert for innovation – buzzword bingo.
    https://www.youtube.com/watch?v=cgeLY7CL5IE
    The Golden Age of NIH seems to be fading in the rear view mirror.

  2. Report this comment

    Michael Andresen said:

    Sadly what is forgotten here is that peer review at its best is a deliberative and iterative process. This is true for scientific publications. Good peer review actually improves, polishes and refines the ideas toward better quality. The two strikes rule really has reduced and degraded peer review at NIH. The second aspect of this rule prohibits submission as a new grant – the same line of work that was rejected. This latter aspect is particularly troubling with the present low funding rates since even a proposal that ranks in the 15% somehow becomes trash that is deemed unworthy to ever be submitted again. This is the poisonous end of a process that does not exist in the scientific peer review of publications. It is explicitly illogical and of course incredibly wasteful and not represented in the simplistic analysis of time to funding. There are other factors such as the decline of stable study section rosters and the plummeting of appropriate experience in Scientific Research Officers. The idea that wait in line is dead really does not agree with experience or conversations with study section members or even human nature. The data displayed of course show that A1s rise in parallel to A0s success – it is a zero sum game – what else could the data show. There simply is not enough money of course. Politics and trendy ideas are displacing solid science and informed review. It is a common experience now that peer review at the best journals provides more appropriate critiques than evaluations from often rashly constructed review panels at NIH. Peer review at NIH is broken. It was never good at picking better than 20%. The NIH hierarchy defensively shops for hollow statistics and PowerPoint rationales that resemble the old IBM advertisement
    for innovation depicted as buzzword bingo.
    https://www.youtube.com/watch?v=cgeLY7CL5IE
    The Golden Age of NIH seems to be fading in the rear view mirror – partly it is the money but the other part is a mindless reinvention of peer review.

    1. Report this comment

      Ally Kanamisa said:

      “Good peer review actually improves, polishes and refines the ideas toward better quality………. what is forgotten here is that peer review at its best is a deliberative and iterative process..”

      I feel a little bit disoriented with the concept of peer review expressed in your comment. It seems to me that peer review, as an integral component of scientific development and progress, has been to examine scientific ideas and/or actual achievements in the context of truly validated precedents and convey whether or not what is proposed represents a true advancement in the field or a new paradigm with the potential for scientific or social breakthroughs.

      I entirely agree that the scientific process is deliberative and iterative and, in most circumstances, “improves, polishes, refines” and validates ideas taking them to higher scientific dimensions. In my view, this is neither the realm nor the essence of peer review. At least, it is not what I have perceived in the history of science and major discoveries.

      I am concerned that your comment reasserts what, virtually all of us, most critizise in reviewers, both in journals and grant reviewing, and that is their inability to look into the essence and specificity of a project and yet proposing “wrapping changes” that might serve their own interests and views.

      1. Report this comment

        Michael Andresen said:

        Not really what I intended. The peers in peer review may offer different points of view even of the same facts. In its most constructive form, these counter views can stimulate alternative strategies toward answers. In standing grant panels in which the proposal returns to at least some of the same people ideally, sometimes panel members can be educated by the give and take engagement of this dialog. What has happened at NIH peer review is that two strikes and you are out makes the judgments more disengaged, bullets of quick and sometimes superficial reactions, and frankly sometimes these can be extremely short sighted. So yes I agree that iterative is essential to “look into the essence” of a project. After all, we should not know the answers at the outset.

Comments are closed.