News blog

NIH answers petitioners, defends grant review policy

Benzra260.jpgThe National Institutes of Health (NIH) has responded to a petition from thousands of unhappy extramural researchers, complaining about a two-year-old policy that prohibits researchers from resubmitting rejected grant proposals more than once.

The response is not likley to satisfy the petitioners. In this letter, sent late yesterday to Robert Benezra (pictured), a cancer biologist at Memorial Sloan Kettering Cancer Center in New York, who instigated the petition last month, top NIH officials make it clear that they are not budging on their two-strikes-and-you’re-out policy for grant submissions.

Since the new policy was put in place in January, 2009, it “certainly has achieved the intended goals,” write Sally Rockey, the agency’s chief of extramural research, and Larry Tabak, its principal deputy director. “The number of applications funded [on first submission] is increasing and there is no queue piling up at the [resubmission] level,” they write.

They add that while “there is little doubt that some great science is not being funded” because a lower proportion of grants overall are being funded in tight budget times, “restoring [second resubmissions] will not change that picture and will increase the time and effort required for writing additional resubmissions.”

Benezra, who drafted the petition in a burst of inspiration while watching the Egyptian revolution unfold on CNN, said today: “I am simply not persuaded.”

Benezra says that in light of current budget constraints, when as few as one in 20 grant applications are successful and the quality of a 20th percentile application cannot be distinguished from a 5th percentile application, “removing the [second resubmission] has the effect of eliminating outstanding grants that would otherwise be funded in time. In effect, queuing is the only fair way to go when dollars are so scarce.”


See here for Benezra’s detailed responses to several graphs provided by NIH, supporting its position. They are are included in NIH’s letter, which Rockey also posted on her website, here.

Benezra initially emailed the petition to 39 colleagues. By the time he submitted it to NIH 11 days later, on February 20, he had 2,335 signatures. Nearly 400 more researchers have since contacted him, asking to sign on.

Benezra said today that he is going to ask for reaction to the NIH response from each of the petitioners and ponder his next move once he has digested their feedback.

He is not without opponents. Comradde PhysioProffe, a blogger who says he is a longtime NIH grantee at a major, private US medical school, was quick to back up the NIH response. In this blog today, he calls Benezra’s argument that great science is going unfunded because of the policy “arithmetically incoherent.”

“The bottom line,” he adds, “is that there are only so many competing awards that can be funded, due to budget constraints.” In the zero-sum game that is NIH extramural grant funding, he notes, reinstituting second resubmissions would necessarily mean rejecting an equivalent number of new submissions, or first resubmissions.

Comradde PhysioProffe put it even more bluntly in this earlier post on the same issue: “These researchers would be much better advised to devote their energies to lobbying Congress to support the NIH budget, not tilting at irrelevant peer review windmills.”

Comments

  1. Report this comment

    Comrade PhysioProf said:

    Just one clarification. I do not consider myself Benezra’s “opponent”, and I agree with him that it is terrible that outstanding science is going unfunded. I simply disagree that the elimination of A2s is contributing to this situation and that the return of A2s would in any way ameliorate it.

    And thanks for the link!

  2. Report this comment

    Paul Knoepfler said:

    If Comradde PhysioProffe is right, which I do not believe, then why allow even one resubmission? Why not only allow new grants?

    It’s just a waste of time to allow people to resubmit even once since only allowing new submissions would (1) fund the same number of total grants and (2) save everyone involved a lot of energy and time.

    Our Comradde ignores the very real stochastic element of grant review and the benefit of feedback given to grant writers by the study section.

    Paul

    http://www.ipscell.com

  3. Report this comment

    Paul Knoepfler said:

    Comrade,

    I think we are all the same side about the unfortunate situation of such low overall funding.

    But based on your argument, why even allow 1 resubmission?

    Aren’t A1’s a waste of everyone’s time just like A2’s were? If you allow A1’s then you have no reason to not allow A2’s. Where do you draw the line and why?

    Paul

  4. Report this comment

    Wafik El-Deiry said:

    If so many scientists agree that A2 applications should be reviewed, why not give the people what they want? They are the peer reviewers too.

  5. Report this comment

    anonymous said:

    I am not persuaded at all that the petition by the “so many scientists” to restore A2s would rescue outstanding grants. The trend, presented in Figure 1 by Rockey in “Early Data in the A2 sunset”, appears to show that the reviewers themselves (since 2009) have been increasingly turning down higher numbers of A2s. If the available funds for biomedical research continue to decrease we might end up, with time, with similar observations for A1s.

  6. Report this comment

    Kornelia Polyak said:

    Any one of you who served on study section knows that it is enough to have 1 out of the 3 reviewers who really does not like the grant (for good reasons or due to poor judgement) to push it in the lower 50%, which means it will not even be discussed. From this score to go to funded level (<10% now), in just one resubmission is almost impossible. NIH (and reviewers on panels) should realize that it is important to score grants based on the applicant. Someone who has been productive and published high quality papers for many years likely to continue to do so and likely to succeed with a project. This is what HHMI and some private foundations do and it seems to work pretty well.

  7. Report this comment

    Eric Guire said:

    I think Benezra has a point that some excellent proposals are scored poorly because of reviewer fallability. The converse is also likely true. Reducing the number of possibile resubmissions decreases the exchange of information between reviewers and applicants, increasing such errors. The limited information exchange also includes insightful feedback from reviewers, which can improve applications and strengthen the quality of NIH funded science.

    To increase information exchange while maintaining review process efficiency, perhaps it would be ideal if PI’s could dilogue with their study sections at some point…what is more efficient than direct, two-way communication? Why not have a multi-stage application process, anologous to that used for screening job candidates? If its the best way to find the right person for a job, isn’t it also the best way to find the right proposals to fund?

  8. Report this comment

    Kate Edwards said:

    I think that “NIH (and reviewers on panels) should realize” that they have the responsibility, before the taxpayers and the scientific community herself, to score grants based on scientific ideas and their potential to better the nation’s health, propel scientific progress and the economics of all people in the US.

    “Productivity and publishing high quality papers” are criteria for evaluating scientific performance and, often but not always, predictors of success in achieving the goals that taxpayers are willing to prioritize at the expense of postponing other national needs. They are not or should not be entitlement insurers.

  9. Report this comment

    Pat Loll said:

    It’s unfortunate that NIH appears to be ignoring the results of peer review of its own peer review system. The community has spoken, but its concerns are being brushed under the rug.

    Culling meritorious proposals at random while strictly limiting the number of resubmissions encourages two bad things: 1) Gamesmanship (to trick NIH into thinking that a new A0 is not actually a rehash of a previous, unsuccessful A1); and 2) Wasted effort and loss of focus (when investigators abandon fruitful areas of endeavor after an unsuccessful A1 and start over in a new area).

  10. Report this comment

    Vinayaka R. Prasad said:

    Correcting for reviewer falliability and providing feedback to the applicant are the two major benefits of a system involving A2s. Both of these are sub-optimal in the current system. There is absolutely no question that the old system was superior.

    System-wide improvements in the review process are hard to achieve without an enormous effort – thus it is understandable that it is enticing to just eliminate A2s in one fell swoop and hope to improve the “funding of the best grants” quickly. But this claim has not really been substantiated. For every grant funded in every study section meeting, it is easy to pick out failed grants that are equally worthy of funding (everyone, who served on a study section lately, knows this). What this is leading to, is for applicants to completely change their direction needlessly and write grants in areas that they really did not intend to – in other words, they may not be putting their best foot forward – leading to that effort just becoming a shot in the dark. Re-writing grants in a totally new direction is a great way to make solid scientists and solid proposals disappear into thin air!!

  11. Report this comment

    Irwin Gelman said:

    Great job, Bob, for bringing this issue to the forefront and for sticking this out. You are correct that they just buzzed past the central argument you were trying to make.

    -They fail to note that the drop in awards to first-time submissions paralleled the drop in the payline from 2003 to 2008. Moreover, the slight increase in this level starting in 2009 probably reflects the ARRA grants, which were all funded as A0s.

    If I’m a Chair of a department with junior faculty applying for their first grant (and I am now), wouldn’t I rather have a young PI whose A1 grant is just shy of funding yet knowing that it will likely be funded on the next resubmission even if it takes another 6-9 months, versus that same young PI whose grant gets a 17th percentile on the A1 and misses funding. Of course I would want the A2 option for this person. Moreover, it becomes nearly impossible to mentor this PI under the second scenario because they are forced to come up afterwards with substantially new Aims on a new submission.

    -As in the 90s, when there was a similar grant funding drought, close to 60% of the young investigators left the system (went to industry or taught gym). The current drought will likely mean that any apparent “increase” in the funding success of new investigators is really because there are fewer left in the system.

    -Tabak’s statements remind me of why our TSA is very successful at accomplishing “security tasks” but making airport travel no safer (e.g., in regards to numbers of hidden weapons getting through checkpoints on test runs). At the end of the day, Tabak can show that he is giving out grants to a group of people with the highest scores, but he can never substantiate that he is funding “the strongest science as early as possible”, even if he programmatically funds a couple extra grants with less competitive scores.

    -If Rockey and Tabak are responsible for overseeing the best system for funding the strongest science, who do they report to, where is the transparency in their system, and who makes these decisions and sets priorities? Are we, the scientific community, not their immediate constituents? And their greater constituents, the American taxpayer, surely want bold research that will better the nation’s health. Given that we all know tales of Nobel laureates whose research was initially trashed by their peers, I believe the American public would rather have important research queue up rather than be flushed at the expense of efficiency.

    -The statement that grants were “rejected” the first time because reviewers knew there were two more tries is beyond ludicrous: we are not allowed as reviewers to consider the “F” word, and moreover, our scores and comments need to match; if they don’t, there typically is a challenge by the Chair or another reviewer. And moreover, there are many occasions when a grant’s score and merits are openly compared to those of another grant (“if you gave that one a 1.4 and this one has better comments, why did you give it a 1.6?”).

  12. Report this comment

    Felix Schweizer said:

    Following the ideas in the rebuttal, I propose that the NIH cancel A1s and have a lottery between grants scored in the top 30%. This would mean that all NIH grants are A0s and are funded quickly. In fact, why not do away with the review system altogether and just make it a lottery? Grants (all A0s) can then be funded in less than a week after submission. (This would also be a ‘green’ solution as it eliminates travel for study section members)

    To put it less cynically, if the review system has any merit – and we probably all hope it does! – then grants should improve through review. Resubmission should thus be encouraged. At how many reviews we should stop is then the important question and not how many A0s get approved.

    Finally, the response still leaves largely unaddressed the question of what is a new proposal and what is a resubmission. Do new proposals really need to be toiling completely new areas or would a tweak in the experimental setup be sufficient? Independent of the question of how many A’s there should be, this dark area is in dire need of clear and bright illumination!

  13. Report this comment

    Peter K Lauf, MD. Professor Emeritus said:

    With great respect I followed how Dr. Benezra tried to take on the Goliath of the US Industry-Science complex, the NIH, to propose, with the support of so many of us, an important and fair change in the review process to include A2 applications. I have supported it.

    For me, a senior investigator, who was funded almost 30 years, there are more fundamental issues at stake, of which the A2 question is only the tip of the iceberg.

    Due to my long career, I have the privilege of a very different view as to how we should do science with the tax-payer’s money. Over the decades, especially since the early nineties, I have seen how the tax dollar accumulated more and more in the hands of a few, just as our overall economically lopsided system has worked to widen the gap between those who have and those who have not. This perpetuation of the Calvinistic-capitalistic system into science to create super-funded groups with multiple RO1 grants on top of merit awards and program project grants, of truly industrial, not academic nature and excluding many folks with equally bright minds but very different value systems, may be a short term success and certainly financial gain. For the future of science and education of younger generations, this short-sighted vision is of catastrophic consequence, as we in the trenches at underfunded state institutions are well aware of.

    We have generated industrial research systems that, in Einstein’s words, “drill the holes at the thinnest part of the wood”, and forgot that training and developing the scientific mind is a lifelong process that requires continuous modest but not gigantic infusions of federal dollars. It is well known that in particular in many private institutions multiple RO1s are used as promotion tools to boost salaries beyond reasonable levels creating a self-regenerating “caste” system of academic standing and purported intellectual ability. This multiple RO1 effect removes our rich colleagues with their chains of postdocs from the realities of being personally involved in the bench training of young scientists, and demonstrating to them, with modest means, that it is indeed worth to choose a scientist’s career.

    I am not asking for socializing research, but rather, as for example in the quest to get the A2s reviewed and the Ao and A1 stats un-skewed, for a fair distribution of the limited resources to permit scientific inquiry at the fundamental level, just as it was once conceived, executed and propagated in past generations (and not with the translational rhetoric that we hear ad nauseam to justify a grant application).

    An opportunistic minority of us teamed up with the leadership of the NIH-industrial complex disregarding the breakdown of the values underlying the formation of a national cadre of trainees determining the future US academic culture and the long term outcome of scientific inquiry. In short, greediness has become the currency, and the sustaining values, ensuring generations of trained folks from our own soil, have been dismissed. Our NIH Scientific Review leadership is out of touch with the catastrophic reality emerging from the abuse of its own freedom once considered a privilege, a freedom that no longer exists and that is so essential for the creative mind. What Dr. Benezra has encountered is a fight against the windmills that have been placed into operation as dictated by the market while funds were abundant and a minority learned together with their leadership to manipulate them. A point in case is Figure 3: only the small percent of Ao applicants, interestingly kinetically indistinguishable from the established folks, gets funded faster.

    I believe, if we do not listen to each other, we will not change an iota our broken system.

  14. Report this comment

    Michael F Romero said:

    NIH funding woes and the loss of A2’s

    Thank you Bob! Great job raising the Discussion.

    There are already many GREAT ideas in this blog; it is had to know where to start. I will try.

    • Everyone agrees that overall NIH funding is a problem. This, however, is an issue to be taken up with Congressional representative and the American tax payers, not the NIH per se.

    • As a former member of an NIH study section, review, discussion and responses are GOOD (as pointed out). The analogy of the Dept. Chair who wishes to advise junior and even senior colleagues is right on target. There is clearly a balance between infinite A’s and A0/A1. Although I typically hate sports analogies, it works here: “3 strikes, and you’re out.” For ~10 years this WAS NIH policy, moving from the previous “infinite resubmission.” Likely there was a bit of a similar outcry in the previous transition, BUT as stated repeatedly, NIH funding was at 25-30% NOT 5-15% (some institutes like NIDDK cut y5 and administratively reduce grant budgets to spread funds and keep ~15%).

    With flat or falling budgets, this is NOT the time to limit review and resubmissions.

    • As also pointed out, data in the NIH letter is misleading.

    “Anonymous” posted 3/27@ 1:00am that “Figure 1 by Rockey in “Early Data in the A2 sunset”, appears to show that the reviewers themselves (since 2009) have been increasingly turning down higher numbers of A2s…” (see above comment). Another bit of mathematical trickery, is that 2009 /2010 show decrease funding rates of A2s. Not to be trite, but “duuuuh.” This coincides with the NIH imposed limit of A2, and so necessarily not allowing A2 submissions decreases the numbers of A2s funded. And of course, this is WHY the % A0/A1’s funded is increasing abruptly in 2009/2010, i.e., ever diminishing levels of A2s.

    • Figure 2 is irrelevant because this has nothing to do with the A2 policy, but rather a simultaneous push to

    a) force Study Section discussion of 75% of ALL New Investigator proposals (GREAT for the New investigator community)

    b) fund new investigators at ~5% pts higher than other established investigators

    • Figure 3 is only slightly meaningful. “Time to Award” has the hidden variable of the underlying changes in NIH policy during the time period. Other policies, i.e., electronic submission, electronic scoring, standard review templates, bullet point of Strengths AND weaknesses (good for the community), have been instrumental in decreasing the time to Award. This internal NIH improved efficiency has in my opinion improved the quality of the reviews, but the BIG impact has been to decrease the “NIH processing time.”

    Everyone wishes that the US was in better financial shape. It would also be nice if more funds could be targeted to BioMedical research to facilitate new drug targets, new therapies and new diagnostics. The present “2-strike” policy and elusive “51% new application” policy at NIH seems counter productive in the present funding climate. Frankly, as a PI, I wish I could spend more time in the lab doing the experiments we have been talking about rather than “begging for funds” to keep the lab afloat. Perhaps someday.

    In the meantime, as pointed out by another, the funding levels are at the level of noise and randomness in the Peer Review System. Perhaps a general lottery to fund research is in order. In any case, a re-instatement of A2’s during this economically challenging time, would seem to be at least a band-aide.

  15. Report this comment

    Michael Chikindas said:

    I should probably start by saying that I was one of those who did sign the petition – not one of these “happy with NIH” people who are closing their eyes on the rotten body of the colossus just because they are “inside the system” and benefit (long time!) from its existence. I should also start by saying that there ARE many decent researchers who are continuously receiving NIH grants and there ARE some decent people working for NIH.

    However, the body of this monster is sick with bureaucracy and often (if not always) operates by the well known algorithm: manus manum lavat. Please, if there is ANYBODY here who thinks that this is not true – you can throw a stone at me, but I doubt there is a single person with such level of “innocent unawareness”. We all know too well what it means when Madam Rockey and Monsignor Tabak are saying that “the policy is working as planned.” After all, how the “policy” may fail if NIH does not obey The Freedom of Information Act? Everybody knows that while we can submit our SPECIFIC complains to NIH, it does not mean that they will be addressed or that NIH will take any actions against its employees – in fact, NIH does not have to inform the complaining person about the outcome of investigation (sic!).

    What kind of analysis is suggested by Madam Wadman? The “Shorter, faster, better?” graph is ridiculous but it serves the purpose: to dilute our attention and to take it away from the real problem. How many of these “first submissions” (top) are by researchers with established connections with NIH – this is the question we must be asking. And – what does the “time to approval mean”? Nothing!

    I quote: "Tabak told Nature that, under the former system, peer reviewers tended to reject meritorious grants that were being submitted for the first time because they knew that the applicants would have two more attempts at funding — in effect setting up a queuing system like that of planes lining up for take-off. The two-strikes policy, he says, “is accomplishing what it set out to do, namely to ensure that we support the strongest science as early as possible”. I wonder if Monsignor Tabak REALLY thinks that we, researchers are THAT naïve to trust his words. What it means, in reality, is that the “queuing system” is even tougher that it was ever before! It simply means that the “outsiders” will have even less chance to be successful in their competition against those favored by the system.

    It looks like Monsignor Tabak and Madame Rockey have some kind of a “magic device” to sample and evaluate pretty much everything! I wonder what their confidence is based on when they are saying that the new policy “seems not to have had an effect on new or early-career investigators”?! The Two Moguls are, in fact, counter-attacking by saying that “returning to a three-strikes policy might in fact work against young scientists trying to launch independent labs by lengthening the time it takes to get winning grants through the system.” Do they really believe in what they are saying?! There is not an ounce of logic in their words! The Truth, however, is very simple. There are not that many ways of getting into the “system”: one has to be a very good scientist AND AT THE SAME TIME be lucky to have his/her proposal reviewed by the panel who UNDERSTANDS the nuts and bolts of the proposal OR to just be “connected”.

    This brings us to yet another flaw of the system: lack of highly professional, knowledgeable, decent, honest, responsible, professionally-ethical reviewers. It happened way too many times to consider it a “single accident” when high quality proposals were hammered by some ignorant “reviewers”, and it did happen too often when the applicant’s cry for help and for justice remained unheard. It is all really very simple: those who are NOT SUPPOSED to review cannot decline the offer because they have two humongous spheres between two body’s extensions which purpose is to provide with mobility, and those who are supposed to select, to control and to provide justice… well, they don’t really care about anything except the calmness of their environment (otherwise why would they go against FOIA?).

    BOTTOM LINE: this is NOT about the number of resubmission, this is about overall health status of NIH as a granting agency – and this status is far from being healthy. It is hideously diseased. And, if you want to learn the truth about NIH, you should read this: M. Chikindas. 2010. New initiative for humanities granting opportunities: how reality differs from “official information”. Journal of Irreproducible Results 51(2): 30-33.

    http://foodsci.rutgers.edu/chikindas/MChikindas%20Manuscripts/JIR-2010-51-30.pdf

  16. Report this comment

    Deepak Bastia said:

    The party line seems to be " CSR does a competent and high quality job in getting grants reviewed and all of the problems that investigators are complaining about can be blamed on the stringent NIH budget. This statement is false. There are two separate issues (i) every one agrees that funding level should be increased by the congress.A 20 percentile funding level would get a lot of the good science supported; (ii) the second problem is that there are a significant number of bogus reviews by incompetent reviewers that are not sufficiently knowledgeable in the field and they lack the humility to inform the SROs about their lack of competence to review applications outside their fields. The procedure to pick reviewers with the right expertise should not be left entirely to the SROs who are probably already overburdened. Careful, fair and competent reviews will significantly ameliorate the situation. The scientific organizations such as ASM and ASBMB could do a better job in their lobbying efforts. Presently, they seem to be too cozy with the NIH.If the NIH bureaucracy does not respond properly to petitions like the one from Dr. Benezra (and this investigator totally agrees with him), that petition should be sent to the congress. Some people fear that such an action could used as an excuse to reduce the NIH budget. There may be some risk involved but the bureaucrats will listen to those who control their budgets. We the tax payers pay the salaries of the bureaucrats. They are our employees.

  17. Report this comment

    Frank Anania said:

    It is unlikely we will ever see the A2 submission return. While the whole system seems to be broken, and grossly underfunded, I tend to agree that the A1 submission is also now a useless effort. In my own recent experience with submission of a competing renewal I got an excellent but not fundable score in this funding environment on the first submission. In the next submission, despite the original reviewers requesting minimal changes, to my surprise the A1 was triaged. In addition to having a new SRO I suspect the grant was not reviewed by the same reviewers. This is particularly the case with so many ad hoc reviewers who pass through for one section meeting and are replaced by others. I have had other colleagues who are superior soientists in my field to whom their competing renewals have been elminated. Productivity seems to have become less important. Some of us have come to the conclusion that the day will come when the competing renewal mechanism will also become extinct.

    The rave of excitement about funding young investigators will come crashing down when they face these hurdles mid-level investigators are now facing.

    There is no way the present trajectory can be creating a positive impact on the future of biomedical research in the US. Very sad indeed.

  18. Report this comment

    Robert Benezra said:

    It looks as though the NIH has finally come to their senses with regards to grant resubmissions, adopting some of the language used in the petition signed by over 3000 scientists in 2011.

    “NIH and AHRQ Announce Updated Policy for Application Submission:

    The National Institutes of Health (NIH) and the Agency for Healthcare Research and Quality (AHRQ) have announced a change in policy on application submissions. Effective immediately, for application due dates after April 16, 2014, following an unsuccessful resubmission (A1) application, applicants may submit the same idea as a new (A0) application for the next appropriate due date. The NIH and AHRQ will not assess the similarity of the science in the new (A0) application to any previously reviewed submission when accepting an application for review. Although a new (A0) application does not allow an introduction or responses to the previous reviews, the NIH and AHRQ encourage applicants to refine and strengthen all application submissions.

    “In this extended period of tight funding, this approach resulted in many meritorious research applications being deemed ineligible for additional submissions, and many investigators having to propose substantial changes to productive research programs.

    New Investigators may have been significantly affected because new research directions may be quite difficult during this phase in their careers.”

    I, and many of my esteemed colleagues are fully aware that this is only a temporary but necessary patch for a system in need of a major overhaul.

Comments are closed.