This is the text of a Correspondence published in Nature (460, 949; 2009) by Conor O’Carroll of the Irish Universities Association:
Your News story ‘Italy outsources peer review to NIH’ (Nature 459, 900; 2009) highlights a problem common to many countries with a small population of research scientists. Ireland can be held up as a successful model in addressing this problem because, over the past eight years, funding agencies have moved to fully international peer review.
A few years ago, important research and development ventures were set up with a new infrastructure to attract talented people from abroad. The use of only Irish peer reviewers to allocate millions of research euros to a small number of universities could not stand up to the principles of objectivity, transparency and perceived fairness and would have led to conflicts of interest. Despite initial opposition, exclusively international review is now accepted; researchers want to be benchmarked internationally as well as nationally.
The typical process for research evaluation in Ireland is to consult four or five reviewers by mail for each proposal. Proposals are then assessed by a panel of invited experts, who meet in Ireland. Reviewers may be sourced through international funding agencies, or by letting applicants nominate experts themselves.
Some Italian scientists in your News story express reservations. They may well have a point, as US reviewers will probably not have any detailed knowledge of how research is conducted in Italy. One approach is to have nationals involved, either as observers or in a formal non-voting role. For example, the Irish Health Research Board organizes international mail reviews and panels, but the chair of each is Irish. They cannot participate in selection, but ensure that the correct procedures are followed and can explain the national research-funding policy. International panel members appreciate this local input, which helps them think outside their own national funding system.
Reviewing criteria often include the quality of the project, the researchers and their institutions, and the social and economic impact of the research. It is important that international reviewers focus on the quality of the first two, as the standing of institutions and the probable impact of a project can be harder for them to evaluate. Also, they should not get involved in detailed budgetary considerations, as these are strictly national.
Things have changed radically in Ireland’s research over the past ten years. In 2008, the country appeared for the first time in a list of ‘Top countries in all fields’ (ranked by citations per paper). We are now placed 19th, up from 36th place in 2003. I believe that international peer review played a significant part in this development.