International peer review improved Irish research rankings

This is the text of a Correspondence published in Nature (460, 949; 2009) by Conor O’Carroll of the Irish Universities Association:

Your News story ‘Italy outsources peer review to NIH(Nature 459, 900; 2009) highlights a problem common to many countries with a small population of research scientists. Ireland can be held up as a successful model in addressing this problem because, over the past eight years, funding agencies have moved to fully international peer review.

A few years ago, important research and development ventures were set up with a new infrastructure to attract talented people from abroad. The use of only Irish peer reviewers to allocate millions of research euros to a small number of universities could not stand up to the principles of objectivity, transparency and perceived fairness and would have led to conflicts of interest. Despite initial opposition, exclusively international review is now accepted; researchers want to be benchmarked internationally as well as nationally.

The typical process for research evaluation in Ireland is to consult four or five reviewers by mail for each proposal. Proposals are then assessed by a panel of invited experts, who meet in Ireland. Reviewers may be sourced through international funding agencies, or by letting applicants nominate experts themselves.

Some Italian scientists in your News story express reservations. They may well have a point, as US reviewers will probably not have any detailed knowledge of how research is conducted in Italy. One approach is to have nationals involved, either as observers or in a formal non-voting role. For example, the Irish Health Research Board organizes international mail reviews and panels, but the chair of each is Irish. They cannot participate in selection, but ensure that the correct procedures are followed and can explain the national research-funding policy. International panel members appreciate this local input, which helps them think outside their own national funding system.

Reviewing criteria often include the quality of the project, the researchers and their institutions, and the social and economic impact of the research. It is important that international reviewers focus on the quality of the first two, as the standing of institutions and the probable impact of a project can be harder for them to evaluate. Also, they should not get involved in detailed budgetary considerations, as these are strictly national.

Things have changed radically in Ireland’s research over the past ten years. In 2008, the country appeared for the first time in a list of ‘Top countries in all fields’ (ranked by citations per paper). We are now placed 19th, up from 36th place in 2003. I believe that international peer review played a significant part in this development.

Stop playing politics with the peer-review process

This text is from a recent Editorial in Nature (460, 667; 2009):

In a depressingly familiar display of irresponsible politicking, the US House of Representatives has taken aim at three studies funded by the National Institutes of Health (NIH). Representative Darrell Issa (Republican, California) introduced an amendment killing the projects on 24 July, during a debate on the NIH’s 2010 budget. The House passed the amendment by a voice vote.

Issa was unhappy that the studies looked at substance abuse and HIV risk behaviour, and that the subjects were outside the United States. One focused on Russian alcoholics, another on female sex workers in China and a third on female and transgender prostitutes in Thailand. All three passed muster with NIH peer reviewers, and together would cost about $5 million over five years. Issa wanted that money to be spent at home, and complained that HIV had been heavily studied already. But his reasoning is specious: alcoholism, prostitution and HIV do not respect borders, and any behavioural information that could help slow the transmission of HIV is crucial. Some 33 million people are infected worldwide, and a vaccine is nowhere in sight.

Issa’s tactic is not new. Since 2003, conservative House Republicans have tried at least five times to strip funding from peer-reviewed projects that drew their ire. Such meddling threatens to undermine the peer-review process as well as potentially eroding the public’s trust that science is above politics.

Also worrying is the House Democrats’ acquiescence to Issa’s amendment. Democrats facing tough re-election bids hoped to dodge Republican attacks in media adverts in their home districts that might have resulted from opposing Issa. Their assumption is that the amendment can be quietly removed when House and Senate negotiators meet to square their versions of the NIH bill before a final vote on it. But Congress should renounce all tactics that undermine peer review — and cease indulging those who use them.

The wait continues for NIH Challenge Grant applicants

From Nature News (Nature 460, 676; 2009), by Meredith Wadman:

Applicants for the coveted Challenge Grants issued by the US National Institutes of Health (NIH) as part of the American Recovery and Reinvestment Act learned the peer-review scores for their proposals late last month. Yet they received little in the way of certainty over whether those scores will translate into money come September, when the NIH will announce which grants it plans to fund.

Competition for the US$1-million, two-year awards is fierce — the agency in Bethesda, Maryland, received more than 21,000 applications, and the NIH director’s office will fund only about 1% of these. With ordinary grants, applicants can usually tell if their grant is fundable as soon they receive their percentile score because they already know the designated ‘payline’, or percentage of fundable applications. The NIH has designated an initial $200 million of $10.4 billion in economic stimulus funds for the grants, but with so many variables at play in allocating the stimulus money, predicting whether a given score will land funding is almost impossible — meaning that those with percentile scores in the mid-single digits are left hanging.

Meanwhile, the burden on the thousands of grant reviewers has, according to some, turned out to be bearable. Gary Johnson, chairman of the pharmacology department at the University of North Carolina, Chapel Hill, told the NIH that he could review up to five Challenge Grant applications. “And they only gave me a couple,” he says. “I don’t know anyone who was overwhelmed by reviewing, because there was an overwhelming agreement of investigators to participate in the process.”

Hoax paper accepted for publication

A story published in Nature News online on 15 June, describes how the editor-in-chief of a journal is to resign after claiming that the publisher, Bentham Science Publishing, accepted a hoax article for publication without his knowledge.

From the Nature News story:

The fake, computer-generated manuscript was submitted to The Open Information Science Journal by Philip Davis, a graduate student in communication sciences at Cornell University in Ithaca, New York, and Kent Anderson, executive director of international business and product development at The New England Journal of Medicine. They produced the paper using software that generates grammatically correct but nonsensical text, and submitted the manuscript under pseudonyms in late January. After receiving several unsolicited invitations by e-mail to submit papers to open-access journals published by Bentham under the author-pays-for-publication model, Davis wanted to test if the publisher would “accept a completely nonsensical manuscript if the authors were willing to pay”. The manuscript was accepted with a request from the publisher for Davis to pay US$800 to its subscriptions department, based in the United Arab Emirates, before the article was published. Davis then retracted the article.

Bambang Parmanto, an information scientist at the University of Pittsburgh, Pennsylvania, and editor-in-chief of The Open Information Science Journal, told Nature that he had not seen the manuscript or any peer review comments before it was accepted. Nor was he informed that the manuscript had been accepted for publication. “I think this is a breach of policy,” he says, adding: “I will definitely resign. Normally I see everything that comes through. I don’t know why I did not see this. I at least need to see the reviewer’s comments.” Parmanto says that Bentham Science Publishing told him that the manuscript had been reviewed by one member of the journal’s editorial board. “The peer review didn’t work,” says Parmanto, who now fears that the journal’s publishing system could be open to abuse. “The publisher could take advantage of the fees, and that is why I want to leave,” he says.

Mahmood Alam, director of publications at Bentham Science Publishing, told Nature in an e-mail statement that “submission of fake manuscripts is a totally unethical activity and must be condemned.” He defended Bentham’s peer review process, saying, “a rigorous peer review process takes place for all articles that are submitted to us for publication. Our standard policy is that at least two positive comments are required from the referees before an article is accepted for publication.” In this particular case, “the paper was reviewed by more than one person”.

Peter Suber, a philosopher at Earlham College in Richmond, Indiana, and a proponent of open-access publishing, is worried that the case could turn people against the author-pays open-access model. “There are many legitimate and rigorous open-access journals that use this same business model,” he says.

Further details are provided in two articles at The Scholarly Kitchen blog: an account of the experiment by Philip Davis, Nonsense for dollars; and an editorial, The tip of an Iceberg, by Kent Anderson.

Janet Young, in a comment to the Nature News story, writes: “I’ve had six requests to review papers from a Bentham Science journal over the last year. The first was for a paper in my field, but I refused as I was very busy at the time. The other five have been for papers in fields I know nothing about – I would have been an utterly inappropriate reviewer had I accepted the requests (I didn’t). Each request had the full paper attached to the email, rather than just an abstract. That seems like a very unusual review process to me.”

See Nature’s news website for the full version of the article, and to add your own comments. You are also welcome to comment here.

No gender bias identified in peer-review of grant applications

NatureJobs reports on the contentious topic of possible gender bias in peer review (Nature 459, 602; 2009). Peer review assesses what is of value in science, yet it has been criticized for biases. One such perceived bias is gender, although evidence for such a bias has been contradictory. A 2007 meta-analysis (L. Bornmann et al. J. Informet. 1, 226–238; 2007; reported at the time in NatureJobs Nature 445, 566; 2007) concluded that women are at a disadvantage in peer review of funding applications. As this study incorporated all known research on this issue, it was suggested to be definitive.

Nevertheless, H. W. Marsh et al. (Am. Psychol. 63, 160–168; 2008) presented conflicting results the following year. This study was comprehensive, based on data from the Australian Research Council (10,023 reviews by 6,233 external assessors of 2,331 proposals from all disciplines), and concluded that the gender of the applicant had no effect on the outcomes of peer review, irrespective of the discipline, gender and nationality of the reviewers, and whether reviewers were selected by a funding panel or chosen by the applicants.

Why should these two studies have conflicting results? To investigate, Marsh and Bornmann now report in NatureJobs how both research teams worked together to reanalyse the data and extend the original meta-analysis. They describe how they applied new, stronger statistical approaches to 66 sets of results representing 353,725 proposals from 8 countries. In this extended study, which will be published in Review of Educational Research, they found no effect of the applicant’s gender on the peer review of their grant proposals. This lack of effect held across country, year of publication of the studies included in the meta-analysis, and disciplines ranging from physical sciences to the humanities.

At least for grant applications, all of the co-authors from each of the research teams agree that the weight of evidence suggests that the applicant’s gender has no effect on the outcome of peer review, and that these findings are robust and broadly generalizable.

Herbert Marsh is a professor of education at the University of Oxford, UK

Lutz Bornmann is a PhD student at the ETH University in Zurich, Switzerland.

This Peer-to-Peer post is an edited version of their NatureJobs article (which is free to access online).

There is some discussion of this NatureJobs article and the timing of its publication at Nautilus blog.

Note: readers interested in the Review of Educational Research paper in advance of its publication can obtain a copy directly from Dr Bornmann or from the NatureJobs editors.

Websites encourage direct public funding for research

The ‘SciFlies’ project, according to a Nature news story (Nature 459, 305; 2009), will profile scientists from a range of disciplines and the new ideas they want to pursue, or ways in which they would like to expand their current research programme. Website visitors will be able to donate any amount to support the projects they find most interesting or worthwhile.

The website itself states: “We look forward to receiving your application for funding of initial proof-of-concept STEM research projects in the range of $5,000 to $12,000. To participate in this unique online grassroots-funded opportunity, please complete the questionnaire about your project, including details of its possible outcome/impact and profiles of the researchers or research team. will then depict the project for online users to view and decide if they want to make a charitable contribution in support of the project-funding goal. Once each project’s funding goal is reached, researchers will be notified and the process of funding will be completed based on a mutually acceptable agreement of terms between researcher and….Projects are competing for support from the general public, so researchers are encouraged to describe their work in appealing and accessible terms, such that users can easily understand the concepts and potential outcomes. Avoid “science-speak,” acronyms or abstract language. A proposal that conveys the researcher’s excitement about a project and its potential, as well as providing insight into his/her personal story, can significantly attract donors.”

At this stage, there is nothing on the website about the peer-review or other assessment process, but there are already some projects listed. A full launch is promised for mid-July.

According to the Nature News story, “David Fries, a marine engineer at the University of South Florida in St Petersburg, conceived of and heads the SciFlies effort. His main inspiration was long-standing frustration with a research funding structure that, with few exceptions, offers scientists no intermediate steps on the way to requesting full grant funding.” Daniel Gaddy, in a comment to the News story, mentions a similar enterprise called FundScience, which also seeks direct public funding of research projects. FundScience provides a description of its activities here, but although a press release is also provided, there is no information about whether a form of independent peer-review will be used to aid potential donors.

Research council amends controversial grant-funding proposal

After a campaign by scientists, the UK Engineering and Physical Sciences Research Council (EPSRC) has softened and delayed its controversial policy to bar serially unsuccessful grant applicants from making funding bids for one year (Nature online News, 5 May 2009). The ban was due to be imposed on 229 researchers starting on 1 June, in an effort to reduce pressure on an overloaded system that currently peer-reviews all grant applications. But eight weeks after it published the policy (see Nature online news 19 March 2009 and Nature 458, 391; 2009), the EPSRC now says that the restriction will not come in until 1 April 2010 — giving scientists more time to change their grant-submission behaviour so that they do not fall under criteria defining repeated failure. And instead of being excluded outright, researchers will be allowed one application during the year.

“We have made these adjustments to address concerns raised by the community — for example, the retrospective nature of [the policy’s] implementation,” the EPSRC said in a statement. “We’ve made bold changes to protect peer review, but we’re not an insensitive organization.”

Peter Main, director of education and science at the Institute of Physics in London, says the EPSRC has listened to criticism and has shown flexibility. “It’s the policy that it perhaps should have been in the first place,” says Joe Sweeney, an organic chemist at the University of Reading, UK, who set up an online petition demanding the policy be repealed, signed by more than 1,900 scientists.

But some researchers say they are disappointed not to have been consulted more directly beforehand — which might have prevented the EPSRC from introducing the ban in the first place. “It’s something of a shame that we had to force them into this policy change,” says Philip Moriarty, a physicist at the University of Nottingham, UK.

The EPSRC is keeping a policy introduced on 1 April, to refuse uninvited resubmissions of failed proposals, which it says will cut 20% of applications submitted for review. The exclusion policy had been expected to cut a further 10%.

The EPSRC says that letters intended to warn individuals in April were never sent. “We are an organization that listens to the community,” says chief executive David Delpy. “If we can make amendments to help researchers whilst ensuring the overall policy is still effective, then that’s in everyone’s interest.”

Not all scientists oppose the proposals, however. In an online comment to this Nature news story, Nial Wheate writes: “I don’t have a problem with the EPSRC policy. Seriously, if you have three grants all ranked in the bottom half in one year, then you need to stop and think about what research you are trying to get funding for. And what’s wrong with only submitting 2 grants per year to make sure this doesn’t happen to you? The policy makes sense to me.”

News story first published online 5 May 2009 Nature doi:10.1038/459020b.

A metric for measuring peer-review performance

A guest post from Willy Aspinall

Department of Earth Sciences, Bristol University, Bristol BS8 1RJ UK.

The Nature Editorial (‘Experts still needed’, Nature 457, 7-8 (2009); free to access online) and Harnad’s related Correspondence item on research performance metrics in the 2008 Research Assessment Exercise (Nature 457, 785; 2009) prompt me to suggest that an additional, complementary metric is needed which would measure the accomplishments of research scientists who act as peer-reviewers for journals.

Good reviewing is very time-consuming and, in some ways, just as challenging as authoring an original research paper; time spent doing this well is time removed from one’s own research work. Indeed, the thoughts and comments of a good referee can often represent a fundamental contribution to the science as well as the quality of a published paper, and this input should be recognized, and measured (the American Geophysical Union regularly celebrates ‘excellence in reviewing’ with citations by its journal editors). It is probably fair to say also that tangible good performance in refereeing usually begets ever more requests to review even more manuscripts, with further incursions on the diligent and proficient scientist’s time.

Perhaps a metric for this essential scientific activity of peer-reviewing might be constructed by summing the number of papers refereed by the individual scientist per year, each review being multiplied by the Impact Factor of the journal concerned. As refereeing is usually a solo activity, a metric for this skill, and for the related professional commitment, would be less prey to the shortcomings of performance measurement associated with metrics that attempt to gauge multi-author citations, for instance. Combining a ‘refereeing metric’ with other citation-related metrics to obtain a more comprehensive performance score for an individual scientist should not be an insuperable problem – and this measure can be pooled, as indicated in the Nature Editorial, with expert evaluation.

Willy Aspinall

Nature Neuroscience experience with peer-review consortium

In 2008, the journal Nature Neuroscience joined a newly created community consortium aimed at making peer review more efficient by allowing reviews to be transferred between consortium journals. In its current (April) issue, the editors look back at their experience with the Neuroscience Peer Review Consortium over the past year (Nature Neuroscience 12, 363; 2009).

Journals in the Neuroscience Peer Review Consortium (NPRC) offer authors whose papers are no longer under consideration at a journal an opportunity to transfer reviews of their manuscipts when submitting their paper to another consortium journal. After a year, Nature Neuroscience‘s experience is similar to that of other journals in the consortium, with only a handful of papers being transferred from Nature Neuroscience to another consortium journal.

Similar to the Nature journals’ transfer system, the NPRC system is voluntary for authors and referees. Editors at one journal only know that a paper was reviewed elsewhere if the author chooses to inform them. At Nature Neuroscience, the editors ask referees for permission to release their identities whenever authors ask for their papers to be transferred to another consortium journal. If a reviewer declines to participate, the reviews (comments to authors only) are transferred anonymously.

All the transfers from Nature Neuroscience to date have been to the Journal of Neuroscience, and represent less than 1% of manuscripts that are eventually rejected after review. However, for the papers that were eventually published in the Journal of Neuroscience, the authors reported that the paper had been expedited. Even in the cases where new referees were solicited, authors felt that transferring the reviews from Nature Neuroscience had saved them time and effort.

No papers have been transferred to Nature Neuroscience from other consortium journals.

The Nature Neuroscience editors ask why so few authors are using the NPRC option. They conclude: “Authors may simply not be aware of NPRC or may not know what journals participate in it. Transfer rates may pick up as more authors learn of the consortium. At Nature Neuroscience, we have noticed an increase in the number of referees that state in comments to the editors whether they wish for their identities to be released to other consortium journals or not, suggesting a growing awareness of the NPRC.

It could also be that there are not that many papers that lend themselves well to this process. Many of our authors who have had papers rejected may prefer to take their chances with new referees at another journal, rather than making substantial revisions in response to the concerns raised by our referees. Certainly, our authors appear to be more conservative when deciding to transfer their reviews, preferentially choosing to utilize the NPRC transfer option when the reviewers reject the paper on conceptual grounds and not for technical reasons.

Another factor that influences the success of the transfer is whether the referees allow the release of their identities to receiving consortium journals. Previous reviews are clearly less useful to the receiving journal if the editors do not know who the reviewers were.”

Nature Neuroscience concludes that it is premature to gauge whether the system truly could save referees, authors and editors substantial time and effort. The editors encourage authors, referees and readers to share their views, either by email or by commenting here.

Going all out for the right to reply

From Nature News, 17 March 2009 (doi 10.1038/458264b):

The Max Planck Society (MPS) in Germany has begun legal proceedings against publishers Wiley International in a dispute over an editorial in the February issue of Human Brain Mapping. The society alleges that the editorial grievously misrepresents it and harms the reputation of one of its scientists. It wants the journal to publish a letter from the society addressing these concerns without delay. Peter Fox, an editor-in-chief at Human Brain Mapping, says that the MPS letter went through normal refereeing processes “in a timely manner”, but says he does not know when it will be published. MPS vice-president Herbert Jäckle, deputized to act for the society in this matter, claims that the journal has unfairly delayed the society’s right to reply. Fox, a neurologist at the University of Texas Health Science Center in San Antonio, accepted the MPS letter on 11 March, two days after the deadline the society set before taking legal action, says he will publish the letter together with a reply that “rebuts Dr Jäckle’s various accusations”.

The row originally centred on a dispute over who owned data gathered in the laboratory of Nikos Logothetis at the Max Planck Institute for Biological Cybernetics in Tübingen. Last spring, Amir Shmuel and David Leopold used the data in a paper for a special issue of Human Brain Mapping on spontaneous brain activity — but Logothetis maintains that the data were used inappropriately because they had been gathered for a different purpose, and wrote to the journal asking for the paper not to be published. It was, however, published in May 2008. Logothetis published a scientific rebuttal in another journal, NeuroImage, in January 2009.

Fox and his fellow editors published an extended editorial analysing the dispute in the framework of ethics in the February 2009 issue of Human Brain Mapping. The MPS says that there are numerous factual errors in the editorial’s account of events. Jäckle is also incensed by the article’s claims that he had given permission to publish the disputed paper. “I was only a mediator in a dispute, not an adjudicator as the editorial claims,” he says. “It is not up to the Max Planck Society to permit publication of anything — that responsibility lies solely with the editors.”

The MPS has posted a response to the February editorial on the website of Logothetis’s laboratory. “The goal of our editorial was to use this conflict to discuss the ethical principles that govern responsible conduct of research and peer review,” says Fox, “and thereby to develop guidelines that would prevent future occurrences of this nature.”