News blog

NIH re-evaluating ‘two strikes’ rule – Updated

The US National Institutes of Health (NIH) is considering reversing a highly unpopular policy that allows its grant applicants only one additional try at winning funding if their application is at first rejected.

Senior leaders at the US$31-billion biomedical agency in Bethesda, Maryland, will decide in the next several weeks whether to abandon a ‘two strikes and you’re out’ policy that was instituted in January 2009 as part of an extensive overhaul of peer review at the agency. Before then, grant-seekers were allowed a third try if a proposal twice failed to pass muster with peer reviewers.

A spokeswoman for the NIH said that the agency cannot comment on whether and when it will return to a three-strikes policy, because the decision is still under active consideration by senior officials.

“It’s about time” that the biomedical agency reconsidered, says Robert Benezra, a cancer biologist at Memorial Sloan-Kettering Cancer Center in New York, who spearheaded a 2011 letter from more than 2,300 scientists to the NIH, asking it to return to the three-strikes system.

The NIH’s rationale for the 2009 change was that the three-strikes rule was causing peer reviewers, either consciously or subconsciously, to favour second and third submissions over first-time proposals, creating, in effect, a queue similar to airplanes circling an airport waiting for a free runway to allow them to land. “Support for meritorious science may be delayed if initial submissions are placed at the end of the queue,” the expert group that reviewed peer review for the NIH concluded in a draft report in 2008 that showed the success rates for first-time applications falling from more than 60% in 1998 to 30% in 2007. (See page 33 of the report for the graph plotting these numbers.)

However, the change generated tremendous push-back from scientists, who have complained of it nearly incessantly to the NIH Office of Extramural Research. But after the agency received the petition from Benezra and his co-signers, it held firm to the two-strikes policy.

Sally Rockey, NIH deputy director for extramural research, published the NIH’s response to Benezra in a blog post, which included a graph that showed first-strike applications for the NIH’s mainstay ‘R01’ grants climbing between 2008 and 2010 as a percentage of all funded R01 grants. The new policy “certainly has achieved the intended goals: the number of applications funded [on the first try] is increasing and there is no queue piling up at the [second try] level,” Rockey wrote.

The critics retorted that, in the absence of third strikes, of course more first-strike applications would be funded; they also noted, ironically, that 100% of successful first-strike applications would be funded if the second strike opportunity were also eliminated.

Benezra and his allies argue that it is impossible for peer reviewers to discriminate between a proposal that scores in the top 10% of applicants and one that scores, say, at 19%. They say that the peer review process, by its nature, is not sufficiently fine-tuned to discriminate at that level.

Thus, they say, the upshot of the two-strikes policy, in an era of stagnant NIH funding and historically low grant-application success rates, is that highly meritorious proposals are randomly culled, and their authors sent back to the drawing board. There, they must craft applications so substantially new that months and years of work in an area often needs to be jettisoned. For younger investigators, the situation is particularly problematic, the critics argue, because less-experienced applicants are on a learning curve in writing grant applications, and because they do not have a body of work to fall back on should a particular proposal be rejected twice.

Some younger investigators say that they are feeling the pressure. “It was really grueling knowing you have to be almost perfect — or as perfect as can be — because you only have one more chance,” says Jill Locke, a 28-year-old postdoctoral student at the University of Pennsylvania in Philadelphia, who just submitted her first NIH grant application: a K01 career development award. (The two-strikes rule applies to career development awards as well as to R01s and many other grant types.)

“I would be very happy if they would reinstate” the third strike, says one 40-year-old cancer biologist at Memorial Sloan-Kettering Cancer Center, who failed last year to win her first R01 grant after two submissions, and who recently submitted a new R01 application. Failing after her second attempt was “just really bad,” she says. “It’s just better if you have one more chance.”

Even the most seasoned investigators have their share of heartburn with the current process, says John Moore, a senior HIV/AIDS scientist at Weill Cornell Medical College in New York. At the National Institute of Allergy and Infectious Diseases, where Moore competes for grants, the current payline for applications from established investigators is 6%.

“I defy any group of reviewers to distinguish in any meaningful way between a 6% and a 12% percentile grant,” says Moore. “If a reviewer doesn’t like how you use semicolons, or is having a bad hair day before the coffee kicks in, or was once threatened by one of your papers, you can fall from 6% to 12% in a flash, without any reflection on the actual merits of the application. That randomness has always been in the system, but its impact is magnified by such a low [success rate]. So, having a third chance literally doubles the chance an applicant can overcome a randomly (slightly) negative comment or two that was sufficient to take the application out of the funding range and into the also-rans.”

 

 CORRECTION: An earlier version of this blog used the term ‘success rate’ in place of ‘payline.’ The terms are related, but not the same. A payline is the percentage of investigator-initiated grant applications being funded by an NIH institute at any point in time. A success rate, by contrast, is the percentage of grants funded by an institute during an entire fiscal year. For example, the success rate for National Institute of Allergy and Infectious Diseasesgrant applications in 2011, the most recent year for which data were available, was 20.2%.

 

 

Comments

  1. Report this comment

    Robert Benezra said:

    Let me be more explicit. The longer the NIH waits to reverse this mindless policy, the more outstanding scientists we will lose. There was not one shred of evidence presented by Sally Rockey or others at the NIH or the blogosphere that made even a modicum of sense. If the response from the NIH was an RO1 application, it would have been triaged. Prove to me that a large number of “new” applications submitted and funded as a result of a “just-missed” A1 is leading to better science, and I will reconsider. I look forward to hearing of the reversal of the two-strike policy very soon.

  2. Report this comment

    Robert Benezra said:

    Let me be more explicit. The longer the NIH waits to reverse this mindless policy, the more outstanding scientists we will lose. There was not one shred of evidence presented by Sally Rockey or others at the NIH or the blogosphere that made even a modicum of sense. If the response from the NIH was an RO1 application, it would have been triaged. Prove to me that a large number of so-called new applications submitted and funded as a result of a just-missed A1 is leading to better science, and I will reconsider. I look forward to hearing of the reversal of the two-strike policy very soon.

  3. Report this comment

    Ben Chapman said:

    I could not agree with Robert more. I suppose it’s possible that there’s a small pocket of people out there who disagree, but I have to believe sentiment across the scientific community is as near universal as could be. For new and early stage investigators, the policy hits particularly hard and is not really offset by New/ESI markups. The funding lines are low already, and even significant and innovative work with a good approach requires grantsmanship skills that come only through experience. I would be curious to know the policy’s impact across career stage, area of research, or institutes.

  4. Report this comment

    Ben Chapman said:

    I could not agree with Robert more. I suppose it is possible that there exists a small pocket of people out there who disagree, but I have to believe sentiment across the scientific community is as near universal as could be. For new and early stage investigators, the policy hits particularly hard and is not really offset by New/ESI markups. The funding lines are low already, and even significant and innovative work with a good approach requires grantsmanship skills that come only through experience. I would be curious to know the impact of the policy across career stage, area of research, or institutes.

  5. Report this comment

    Ally Kanamisa said:

    “Prove to me that a large number of so-called new applications submitted and funded as a result of a just-missed A1 is leading to better science”

    Not feasible. It is impossible to prove whether the “two strikes” rule is leading to better science or not because there has not been sufficient time to evaluate the science produced (better or worse). Nor it is possible, for the very same reason, to prove that we are losing more outstanding scientists because of the policy. In contrast, we have very hard evidence of losing many outstanding scientists during the “three strikes” period when we were having more available resources. And we lost many outstanding scientists because the system allowed and favored concentration of resources in a few (some of them outstanding scientists, others with much less excellent or even mediocre science). Unfortunately, the NIH has not been able yet to deal with this problem effectively but is surrendering to the pressure. Let’s be serious and let the experiment running for at least 3 more years and then evaluate it.

    Yes, the universal agreement is that it is virtually impossible to meaningfully distinguish between a 6% and a 12% grant. That the inherent subjectivity of a review could be overcome with a third chance might be as objective as believing in “miracles in science”.

    1. Report this comment

      Robert Benezra said:

      That is precisely the point, it cannot be proven, not now, not ever. So therefore you must adopt the policy that is most fair. It is most fair to let people who have written grants that are unfunded but indistinguishable from the funded ones at the level of peer review, to wait in line (in my opinion for however long they want). One senior NIH official suggested in conversation to me that we just have a lottery for the grants within a certain percentile range which is just as equitable a solution. But you can enter the lottery as many times as you like as long as you can hold out. Why kick people off a food line (ever) when times are tough unless they did something wrong (like write a poor grant by any standard). The failure to appreciate this simple logic is what got us into this situation and it is now time to reverse it.

      1. Report this comment

        Ally Kanamisa said:

        Yes Robert, I heard the lottery analogy from 2 NIH senior officials eight years ago. The lottery element has always existed and will continue to exist. The question arises as to what are the experiments that will provide compelling and INDEPENDENT information on how to minimize, if not eliminate, undesirable lottery effects (e.g., scientific stagnation, unfair and/or entitlement-based use of limited resources, diminished opportunities for future science/scientists etc).

        NIH, as manager of tax-payer resources, has the obligation to be in constant dialogue with the community to advance solutions for the best management of public investments. However, the community will never be uniform on needs and/or expectations. Surrendering to the pressure of specific groups by aborting experiments (for not compelling reasons and ahead of time) that had been designed to implement priorities (i.e. best science and best projects be funded as soon as possible) is not consistent with the nature of an organization whose mission is to use and create science for the common good. At times, the pursuit of the common good requires temporary persistence in unpopular and/or extraordinary measures. That might be the secret of excellence in government and governance. I wish NIH had the strength to persist in finding out if the two strikes rules is of greater benefit than the old 3 strikes policy.

        I perfectly understand your points but I happen to disagree with some of them.

        Take care

        Ally

Comments are closed.