Statistics reveal hint of bias in NIH peer review
From Nature News in Brief (454, 564; 2008):
The system used by the US National Institutes of Health (NIH) to evaluate grant proposals does not adequately compensate for reviewer bias, affecting one in four proposals, a study finds.
Valen Johnson, a biostatistician at the University of Texas M.D. Anderson Cancer Center in Houston, evaluated reviews for nearly 19,000 grant proposals performed by around 14,000 reviewers in 2005 (V. E. Johnson Proc. Natl Acad. Sci. USA doi:10.1073/pnas.0804538105; 2008).
Each application is typically read by 2–5 reviewers, then discussed when a larger study section of about 30 reviewers meets. In the end, scores from all the study section's members — readers and nonreaders alike — are averaged together.
The system fails to account for individual bias and places undue weight on panel members who have not read the proposals, Johnson argues. He found that the top grants were largely unaffected by reader bias, but that it did affect grants closer to the funding cut-off line. Overall, accounting for reader bias changed about 25% of the funding decisions, meaning that one in four funded proposals would have been replaced by one that had not been funded.
See here for a longer version of this story, which includes an informative online readers' discussion.
