News blog

Merit review under the microscope in US

Brooks(Str260).jpg

A US House subcommittee today held a hearing to determine whether the National Science Foundation (NSF) merit review process — by which agency selects the research projects it will fund each year — is identifying the best science possible to support. The subcommittee heard testimony from four witnesses about the quality of selected research and solicited suggestions for how to improve the process.

“This subcommittee must ensure that federal dollars are being spent on the best science,” said Rep. Mo Brooks (R-AL, pictured), who chaired the hearing, held by the House Science, Space and Technology Subcommittee on Research and Science Education. “We want to know if the current process spurs or stifles innovation, … and if there are flaws in the system that may be providing precious federal funds to lower rated proposals over more highly rated proposals.”


Brooks cited figures from FY 2010 data suggesting that proposals receiving low ratings from external reviewers were funded, while thousands that received excellent ratings went unfunded. He asked the NSF’s deputy director Cora Marrett why that was the case.

Marrett responded that ratings were only recommendations and not final decisions on the value of a project. Program officers that make the final decisions take into account a number of other factors, she said, including the potential contributions to innovation and a balanced NSF research portfolio. “All of these can brought to bear as assessments are made about how to use the funding in the most effective ways for the nation,” she said.

Rep. Larry Bucshon (R-IN) expressed concern about young researchers’ chances at being funded in highly competitive, small academic fields, where reviewers may share an exclusive, almost familial mentality. “How do we guard against the politics involved in this process?” he asked. “Is there a review process for the reviewers?”

Marrett pointed to precautions put in place to prevent political influences, including conflict of interest forms that must be signed by reviewers, ethics and conflict training for program officers, as well as a requirement for institutions to declare that there is no conflict of interest. “We have in place checks and balances to reduce the likelihood that there will be the kind of personal considerations that would be a concern,” she said.

Other witnesses suggested ways of streamlining the NSF process and reducing strain on the system, which is receiving increasing numbers of applications each year. In 2010, more than 55,000 proposals were submitted by scientists from across the US, and less than a quarter — about 13,000 — were funded. It took nearly 46,000 members of the scientific community to review those applications, according to Marrett.

Nancy Jackson, president of the American Chemical Society, suggested that program officers should triage proposals upon receipt and reject immediately the 50% or so that are of too low quality to be funded. The measure would allow reviewers to focus on competitive, high-quality grants, she said.

Keith Yamamoto vice chancellor for research at the University of California, San Francisco suggested that reviewers and applicants be required to focus only on the scientific merit of a project rather than whether or not a project will achieve “national goals” as specified under the “broader impacts” criterion in the NSF review process.

“Asking merit reviewers to make an assessment of how well a project fits national goals is inappropriate,” he said, adding that such a judgment should be left to the funding agency. “You’re really asking reviewers to make guesses.”

Such suggestions may be addressed in two upcoming meetings about NSF’s merit review process taking place on 28 July, as the National Science Board (NSB) reviews and revises its process and merit review criteria, including the “broader impacts” review criterion. The Advisory Committee on Merit Review Process (MRPAC) will hold a virtual meeting to discuss possible improvements to the process. At the same time, the Task Force on Merit Review will meet at this week’s National Science Board meeting to discuss the evaluation criteria.

A webcast of today’s hearing can be viewed here.

Comments

  1. Report this comment

    anna lambert said:

    Our country is already behind others on technology and there is a gap on manufacturing technology. But other societies in the past had made these same mistakes of limiting technical growth and so thoses societies where distroyed and weather away their greatness will America follow those same past mistakes. Mistakes like those made by the leaders of Rome, or the Christian societies or controled what knowledge was taught to their society.

Comments are closed.