Peer-to-peer

Peer review is the cornerstone of scientific publishing. But it isn't always clear exactly what Nature Physics expects of its referees. The journal explains in its November Editorial (5, 775; 2009). "Whatever you think about a paper, it is vital to explain to us exactly why you think it. Your colleagues among the other reviewers may disagree with your assessment, and we do not base our decisions on a show of hands. Hence detailed critiques carry more weight in informing our decisions than terse affirmations one way or the other (in most cases we would disregard the latter, regardless of who supplied it). A further point to consider is whether the work presented in a paper is similar to what has been done before — in such a case, please explain exactly what has been done previously and indicate where it was published." The Editorial outlines the journal's peer-review process, what the editors look for in a review, how to write the review, and how the editors make their decisions. "Peer review is essential for maintaining the integrity of the scientific record. It's well worth the effort. And we thank all of you who make it."

Posted by Maxine Clarke on November 06, 2009
Categories: Journals | Permalink | Comments (0)

Perceived lapses in the peer-review process often receive a lot of attention, but the majority of researchers declare themselves satisfied with the system even though they would like to improve it. If it is imperfect or broken, how do we fix it? This question is addressed in the November Editorial of Nature Chemistry ( 1, 585; 2009), in light of some blog commentaries which identified prior publications that had not been referenced in a journal paper.
Open peer-review experiments have generally not been very successful because reviewers are less likely to make forthright comments in an open forum. Double-blind peer review is another option, but one must consider the role of the editor who oversees the process, as well as the difficulties of effectively hiding the identity of authors in smaller fields from other experts — especially when many authors regularly cite and discuss their previous work. The Editorial concludes:
"The Royal Society of Chemistry's Dalton and Faraday discussion meetings provide a unique mix of traditional peer review coupled with both comment (by peers) and responses from the authors, but require members of a particular research community to assemble at a conference. It is in some ways similar to the grant proposal review process at, for example, the US National Institutes of Health. However, such a process is clearly not a viable option for every one of the vast number of papers submitted for publication. The journal Atmospheric Chemistry and Physics uses a system in which, after initial assessment by an associate editor, manuscripts are posted online for comment. After referee reports are received, these are also posted online with the manuscript along with author rebuttals. If eventually accepted, a paper is formally published in the journal, whereas those that are not remain available (and citable) as online 'discussions'. This differs from the preprint servers Nature Precedings and arXiv because there is an initial assessment of the suitability of the work (based on more than just scope).
Perhaps a hybrid system could be the solution. Traditional peer review, and a decision to publish, could be followed by a fixed period in which any interested party could post questions or comments and the authors are given the opportunity to respond — all moderated by an editor — before a final version of the article (including comments and responses) is preserved for the record. This would again require a large change in the habits of the community — authors, reviewers and publishers — and previous experiments with commenting on published papers have been far from conclusive."
Sense about Science peer-review survey 2009.
Nature journals' peer review policy and Editorials on the subject.

Posted by Maxine Clarke on October 26, 2009
Categories: Systems | Permalink | Comments (0)

This post is by Myles Allen of the University of Oxford:
As a vocal supporter of the traditional system of scientists communicating through peer-reviewed channels ( Nat. Geosci. 1, 209; 2008 and associated debate at Peer-to-Peer), I was hesitant about writing a critical Commentary on the Feature 'A safe operating space for humanity' by Johan Rockström et al in the 24 September issue of Nature (Nature 461, 472-475; 2009) in a non-peer-reviewed forum. The Nature and Nature Reports Climate Change editors had clearly thought through this argument: the Feature was not itself peer-reviewed, so no golden rules would be broken in publishing a series of commentaries alongside it in Nature Reports Climate Change.
The problem is that packing the point into a few hundred words, and slipping into the usual bloggers' trap of feeling you have to shout loudly on the internet or no-one will listen, means that the end result reads very black-and-white. Corresponding with Johan Rockström over the past week, it emerges we agree on far more than the tone of my Commentary, 'Tangible targets are critical' (Nat. Rep. Climate Change doi:10.1038/climate.2009.95), probably implies. I understand that Rockström et al. had not originally intended to make the link between a six-degree climate sensitivity, the two degrees target and 350 p.p.m. a focal point (as I read it) of their Feature. Likewise, they observe, reasonably enough, that limiting cumulative carbon dioxide emissions to one trillion tonnes of carbon is just another way of framing the climate boundary, with (as I acknowledge) remarkably similar implications to 350 p.p.m.
If this had been done the old-fashioned way, Rockström et al. would almost certainly have had to qualify their reliance on a six-degree climate sensitivity in the course of the usual to-and-fro with referees. And I in turn would have toned down a lot of my objections. The end result would undoubtedly have been blander on both sides, but would that really have been much of a loss? Following what was, for me, an experiment, I still feel it is very much an open question whether scientific communication in general benefits from direct publication rather than allowing rough edges to be smoothed off through traditional peer-review.
Myles Allen is at the University of Oxford, Parks Road, Oxford, OX1 3PU, UK.
-----------------------------
Context:
Johan Rockström and his co-authors argue in a Nature Feature that to avoid catastrophic environmental change, humanity must stay within defined 'planetary boundaries' for a range of essential Earth-system processes. If one boundary is transgressed, then safe levels for other processes could also be under serious risk, they caution. Seven expert commentaries respond to this proposal in Nature Reports Climate Change, one of which is by Myles Allen. All these articles can be accessed from this index page.
There is an associated Nature podcast in which Johan Rockström is interviewed and editor Ehsan Masood provides further analysis.
Further reading:
A longer paper upon which the Nature Feature is based, 'Tipping towards the unknown', is available at the Stockholm Resilience Centre.
Nature news special: The road to Copenhagen.

Posted by Maxine Clarke on September 23, 2009
Categories: Quality and value | Permalink | Comments (3)

Manuscript peer reviewing is at the heart of the scientific system, but it seems that these duties are often not properly (if at all) recognized by universities, funding agencies or even the rest of the scientific community. This is the main message of the September Editorial in Nature Structural & Molecular Biology, 'The Unsung Reviewer' (16, 899; 2009) The Editorial notes:
"Scientists wear many hats these days. They prepare and teach classes and sit on various committees. Then there are the multiple activities directly related to their research, including grant writing, mentoring students and postdocs, attending conferences, writing papers and reviewing manuscripts. All these duties can take a considerable amount of time and effort and most are recognized as worthy contributions by funding agencies, universities and research institutions when evaluating a scientist's performance. On the other hand, peer reviewing papers seems to be the Rodney Dangerfield ("I get no respect!") of a scientist's duties."
A new report by the Science for Policy project of the US Bipartisan Policy Center notes that peer-review is essential for the scientific system. Journals such as NSMB (and the other Nature journals) would not agree from their own perspective with one of the report's conclusions, that "peer-review is no longer assumed to be a professional obligation". As the Editorial points out, our journals have very broad reviewer pools (running into many tens of thousands for the heavily oversubscribed weekly title Nature, for example), and the editors know and very much appreciate the work that these reviewers put into improving submitted manuscripts.
The SPP report also identifies a need to increase the number of scientists who participate in peer review for federal agencies, making suggestions such as listing such service on grant applications or even making it a requirement for funding. In addition, it urges journals to run a quality system, for example by providing the peer-reviewers with feedback (which the Nature journals do, and which is appreciated by our reviewers - for one reason, because a reviewer can see his or her report in the context of those of the other reviwers of the manuscript). The NSMB Editorial concludes:
"Peer review is often compared to jury duty, a chore that one has to do once in a while as a service to the community. But reviewing manuscripts can be enjoyable and useful at any career stage. True, it takes time, but as one reviewer aptly put it: "I still learn do's and don'ts from reviewing papers, as well as just getting useful information, so it is time spent pretty well." It is also true that one does not get much open recognition from it, and there are limitations on what the journals can do without compromising the reviewers' anonymity. We have in the past publicly acknowledged our reviewers at the end of the year, and will do so again this year. The SPP report stresses that it is important to stress that "peer reviewing manuscripts should be an expected and appreciated aspect of a scientist's career." Of course, we do have a vested interest in this, but we firmly believe that it's important to cultivate a vibrant 'reviewer culture'."

Nature journals' peer-review policies and guidelines.
More on peer review from the Nature journals.

Posted by Maxine Clarke on September 14, 2009
Categories: Quality and value | Permalink | Comments (0)

We received a question at the Nature Network 'Ask the Editor' forum which I thought readers of this blog might find useful.

Q: I once reviewed several manuscripts for a good journal, but it is on behalf of my Ph.D boss. Now, I am a postdoc and wonder how to become a reviewer? Thanks for any suggestions.

A. At Nature, we ask our peer-reviewers to identify anyone who helps them with their review. (The reviewers promise to keep the ms confidential, in advance of being sent it, and undertake to ensure that anyone they show it to also keeps it confidential). Therefore, we often discover (and regularly use) good new reviewers by this method. Many senior reviewers ask junior colleagues to review a ms as part of their mentoring, and are very good at assigning credit to these junior colleagues.
I suggest that if you help your boss or another colleague with his or her reviews in future, you ask that person to name you as a collaborator when he/she sends the journal the review, so your name gets known.
If you have been a co-author on your boss’s papers, you could contact the journal that published the work and offer to review (providing the information that you’ve published in that journal).
I think that as you publish in your own right, journal editors will get to know of you and start to ask you to review for them.
If you meet any journal editors at conferences you attend, you could let them know that you’d be interested in peer-reviewing for their journal.
Good luck!

Posted by Maxine Clarke on September 07, 2009
Categories: Journals, Peer Review Feedback | Permalink | Comments (0)

This is the text of a Correspondence published in Nature (460, 949; 2009) by Conor O'Carroll of the Irish Universities Association:

Your News story 'Italy outsources peer review to NIH' (Nature 459, 900; 2009) highlights a problem common to many countries with a small population of research scientists. Ireland can be held up as a successful model in addressing this problem because, over the past eight years, funding agencies have moved to fully international peer review.
A few years ago, important research and development ventures were set up with a new infrastructure to attract talented people from abroad. The use of only Irish peer reviewers to allocate millions of research euros to a small number of universities could not stand up to the principles of objectivity, transparency and perceived fairness and would have led to conflicts of interest. Despite initial opposition, exclusively international review is now accepted; researchers want to be benchmarked internationally as well as nationally.
The typical process for research evaluation in Ireland is to consult four or five reviewers by mail for each proposal. Proposals are then assessed by a panel of invited experts, who meet in Ireland. Reviewers may be sourced through international funding agencies, or by letting applicants nominate experts themselves.
Some Italian scientists in your News story express reservations. They may well have a point, as US reviewers will probably not have any detailed knowledge of how research is conducted in Italy. One approach is to have nationals involved, either as observers or in a formal non-voting role. For example, the Irish Health Research Board organizes international mail reviews and panels, but the chair of each is Irish. They cannot participate in selection, but ensure that the correct procedures are followed and can explain the national research-funding policy. International panel members appreciate this local input, which helps them think outside their own national funding system.
Reviewing criteria often include the quality of the project, the researchers and their institutions, and the social and economic impact of the research. It is important that international reviewers focus on the quality of the first two, as the standing of institutions and the probable impact of a project can be harder for them to evaluate. Also, they should not get involved in detailed budgetary considerations, as these are strictly national.
Things have changed radically in Ireland's research over the past ten years. In 2008, the country appeared for the first time in a list of 'Top countries in all fields' (ranked by citations per paper). We are now placed 19th, up from 36th place in 2003. I believe that international peer review played a significant part in this development.

Posted by Maxine Clarke on September 03, 2009
Categories: Policy and Public | Permalink | Comments (0)

This text is from a recent Editorial in Nature (460, 667; 2009):

In a depressingly familiar display of irresponsible politicking, the US House of Representatives has taken aim at three studies funded by the National Institutes of Health (NIH). Representative Darrell Issa (Republican, California) introduced an amendment killing the projects on 24 July, during a debate on the NIH's 2010 budget. The House passed the amendment by a voice vote.
Issa was unhappy that the studies looked at substance abuse and HIV risk behaviour, and that the subjects were outside the United States. One focused on Russian alcoholics, another on female sex workers in China and a third on female and transgender prostitutes in Thailand. All three passed muster with NIH peer reviewers, and together would cost about $5 million over five years. Issa wanted that money to be spent at home, and complained that HIV had been heavily studied already. But his reasoning is specious: alcoholism, prostitution and HIV do not respect borders, and any behavioural information that could help slow the transmission of HIV is crucial. Some 33 million people are infected worldwide, and a vaccine is nowhere in sight.
Issa's tactic is not new. Since 2003, conservative House Republicans have tried at least five times to strip funding from peer-reviewed projects that drew their ire. Such meddling threatens to undermine the peer-review process as well as potentially eroding the public's trust that science is above politics.
Also worrying is the House Democrats' acquiescence to Issa's amendment. Democrats facing tough re-election bids hoped to dodge Republican attacks in media adverts in their home districts that might have resulted from opposing Issa. Their assumption is that the amendment can be quietly removed when House and Senate negotiators meet to square their versions of the NIH bill before a final vote on it. But Congress should renounce all tactics that undermine peer review — and cease indulging those who use them.

Posted by Maxine Clarke on August 26, 2009
Categories: Policy and Public | Permalink | Comments (0)