Peer-to-peer

This post is a continuation of the discussion about blogging and peer-review by selected reactions at RealClimate (a climate scientists' community blog) to Nature Geoscience's two commentaries on blogging.

Philip Machanick writes that the "problem with blogs is that there is no way for an outsider to know which are reasonably careful creations of informed scientists, which are opinions of the scientifically illiterate, and which are astroturf creations designed to confuse critics of science that is in conflict with an industry. Given that terrain, I would rather have something like RealClimate than not: it helps to balance things out. Errors tend to be corrected quickly here as a consequence of a large informed readership (even if it is sometimes annoying that you get drive-by ignoramuses who don’t benefit from getting their misconceptions answered). He goes on to suggest that a site such as RealClimate is of value in providing a forum for rebuttals of peer-reviewed science in order to develop a consensus on whether a "formally published rebuttal is worth the effort".

Ray Ladbury's view: "When it comes to peer reviewed papers, one has to presume the reader will have a minimum level of familiarity with the subject matter. One also presumes that the reader will have a day job, and so the question becomes whether the information in the paper is of sufficient interest to the average scientist in the community to say, “Hey, take a look at this. It looks mostly correct to me and has some interesting information/insights/methods…” This is not in any way the gold standard in the sciences. The gold standard comes when the community as a whole says, “Hey, cool, I can use this.” The paper is cited. The techniques are used. Science advances. Eventually, what was in the paper becomes part of the tacit knowledge assumed by reviewers.
The tacit knowledge one can presume for a blog like Realclimate is much lower. One presumes there is an interest in the subject–why else would the reader be perusing the blog. One presumes at least a passing acquaintance with the scientific method and maybe some familiarity with basic results like conservation of energy, etc. One could perhaps assume that the average reader has taken the time to acquaint him- or herself with material to which one is vectored via the “Start Here” button–although this is far from Universal.
For the average newspaper reader of a science story, the tacit knowledge is nearly nonexistent–or worse, wrong. And then we have the blogosphere, where information density is at best, rarified and often toxic.......in an information economy, it seems that all too many readers and journalists are content to remain paupers."

Posted by Maxine Clarke on April 29, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)

In a comment to an earlier Peer to Peer post, 'Regular Scientist' takes issue with Nature Cell Biology's definition of a good report, writing (slightly edited for length): the journal sometimes asks the authors to undertake many experiments even before sending a paper to review. Not strangely, papers in Nat Cell Biology, even small reports, contain many Supplementary Figures (up to 20, I´ve seen). I miss the old days where publications were for sending a very interesting result so you could extend it and discuss its importance for your field. Nat Cell Biology's system is undermining research efforts in the authors´ labs, which have to dedicate an enormous effort to provide many additional experiments that don´t even add much to the concept. This is not what science, and the peer-review process, should be about. I would recommend the editors of Nat Cell Biol a calm reading of the original paper on the structure of nucleic acids, and note the way it was written.

Bernd Pulverer, Chief Editor of Nature Cell Biology, has kindly agreed to my request to write a guest post to respond, and to elaborate on the journal's practices and policies:

As ‘Regular scientist’ points out, we do strive to publish rather well-developed studies in Nature Cell Biology. It is interesting to read the concern about the supplementary information, since we usually hear the opposite complaint: namely that the tight format restrictions applied preclude presentation of complete datasets. In considering submissions, we expect fewer details of molecular mechanistic analysis if the paper reports a highly novel observation. And because the study of cell biology tends to throw up surprising new findings that deserve a wide audience but that cannot be developed in a reasonable timeframe, we decided some years ago to publish a short ‘Brief Communication’ format, which I hope is appreciated by Regular Scientist.
In the Editorial under discussion, we wrote:

Nevertheless, the most important part of the report is assessment of the data: are key experiments or crucial controls missing? Are the data significant and definitive? Are all claims made supported by the data? A black and white model should not be a requirement — in fact it often underestimates biological complexity. Formulate a clear set of recommendations for additions or improvements. Experimental suggestions are important even if an outright rejection is recommended, as it makes for a transparent and constructive report that will allow the author to improve the dataset and select an appropriate target journal. If the overall assessment is positive, do not feel obliged to ask for non-essential experiments. New experiments should add key information, buttress claims or improve existing data, and they must be achievable. Indeed, the perception that referees invariably ask for more data encourages some authors to hold back data in the hope that these will be requested and can then be provided by return post; clearly, this is a futile exercise. Referees and editors alike should take great care that all key issues are raised in the first assessment — draw a line and adhere to it. Subsequent evaluations may raise issues on new data, but raising yet more ways to develop a study is not fair.

We, the Nature Cell Biology editors, agree that one can always ask for more data, and that sometimes data require an unrealistic amount of effort for the information added. However, we entirely disagree that our editorial and review process does not result in the publication of much-improved datasets. The tendency of referees to feel an obligation to ask for some new data, and conversely for authors to hold back data to give referees something obvious to ask for, is an issue that we tried to address in the Editorial.
We certainly do not ask for more data because we, the editors, feel under any compulsion to do so: the simple fact is that a good number of manuscripts submitted to the journal are rather premature, and we cannot justify wasting our referee’s time when we perceive obvious holes in a dataset that the referees would invevitably want to be addressed. We believe that this step is in an author’s interest, since we will not undertake multiple rounds of peer-review without good reason. If requested data are deemed to be unobtainable for technical reasons, or indeed unnecessary, we are always willing to hear an author’s arguments.
Regarding the 1953 Nature paper that Regular Scientist recommends: indeed, the paper is a good read. However, science has become much more complex since then, and the discoveries often more detail-oriented, requiring a more in-depth analysis. At the same time, the tools available have developed tremendously in 55 years. Even in 1953, readers will note that Watson and Crick's conclusions depended on the two following papers in the issue, providing experimental details. The challenge is to set ‘the bar’ correctly to what is achievable and what should be achieved for publication in a top-flight journal. This is what the job of an editor is all about.

Posted by Maxine Clarke on April 23, 2008
Categories: Quality and value | Permalink | Comments (1)

Continuing the discussion of the relative contributions of blogs and the peer-reviewed literature to scientific understanding, I'm highlighting another reaction to the two Nature Geoscience Commentaries presenting different perspectives.
In the comment thread at Real Climate blog, Simon Donner of the University of British Columbia writes:

"Blogs have created a forum for many people to informally discuss science. They are also a great forum for scientists to provide context to their own work and to the work of others in their field.
And some of the time, that may take the form of criticism. That’s ok. But we do have to be careful in blurring the line too much between peer-reviewed publications and blogs. The first problem is the obvious one. Blog posts are unfiltered, un-reviewed, and often written off the cuff, while journal articles are screened, reviewed, and (should be) meticulously researched. It is far easier to write a criticism of a paper on one’s blog than to write a response and submit it to the journal. The issue isn’t just that no reviewers check the work… the blog author is unlikely to do even close to the amount of research and analysis nor give the wording nearly the same level of consideration as is expected in a paper submitted to a high-quality, peer-reviewed journal.
The other problem is that blog posts are readily accessible to anyone at anytime, both in language, and in the unlicensed nature of the internet. To a huge swath of the public, blog posts are the face of science. Like it or not, bloggers with scientific credentials are like self-appointed ambassadors for science. If we are going to write about our science, we should do it with thought, and we should do it well. That is a standard that myself and many other science bloggers often struggle to meet."

Posted by Maxine Clarke on April 16, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)

From Nature 452; 677; 2008:
A federal magistrate in Massachusetts last week ruled that The New England Journal of Medicine (NEJM) does not have to comply with a subpoena issued by Pfizer forcing the journal to provide confidential peer-review documents related to the painkillers Celebrex (celecoxib) and Bextra (valdecoxib).
The drug firm had tried to compel the journal to hand over peer reviews and internal editorial discussions for 11 papers on the painkillers. It argued that these would help it defend the arthritis drugs in lawsuits alleging that they caused heart attacks and strokes (see Nature 452, 6–7 ; 2008 and this post on Spoonful of Medicine blog).
In his 12-page opinion, Leo Sorokin wrote that the material Pfizer sought seemed relevant on first examination, but that “NEJM's interest in maintaining the confidentiality of the peer-review process is a very significant one … and tip[s] the scales in favor of the NEJM" .
The judgement comes three weeks after an Illinois judge ruled against Pfizer after it issued almost identical subpoenas to The Journal of the American Medical Association and Archives of Internal Medicine.
The Nature journals' guidelines for reviewers state: "Reviewers should be aware that it is our policy to keep their names confidential, and that we do our utmost to ensure this confidentiality. Under normal circumstances, blind peer-review is protected from legislation. We cannot, however, guarantee to maintain this confidentiality in the face of a successful legal action to disclose identity in the event of a reviewer having written personally derogatory comments about the authors in his or her reports. For this reason as well as for reasons of standard professional courtesy, we request reviewers to refrain from personally negative comments about the authors of submitted manuscripts. Frank comments about the scientific content of the manuscripts, however, are strongly encouraged by the editors."
We are advised that if peer-reviewers follow this advice, it would be extremely unlikely that there could be legal grounds to force their identities to be revealed.

Posted by Maxine Clarke on April 11, 2008
Categories: Ethics, Policy and Public | Permalink | Comments (0)

The two Nature Geoscience Commentaries expressing opposing views on blogging's role in science research communication have been much discussed in the blogosphere (see Climate Feedback blog, for example.). One discussion took place at RealClimate blog, where Gavin Schmidt's post about his Nature Geoscience view on how science blogs and traditional peer-review intersect has attracted more than 100 online comments from climate scientists and others.
One commenter, "Tamino", writes: "Blogs don’t serve very well for communication among scientists. Peer review does more than just protect us from being inundated with substandard work; it protects authors from their own mistakes and improves the quality of what we write. Peer review itself is an immensely valuable avenue of communication; who among us hasn’t at some time included a phrase like “We thank an anonymous referee for comments and suggestions which dramatically improved the final manuscript”?
But as bad as blogs are for actual research, peer-reviewed journals are far worse for communicating with and educating the lay reader. Yet when it comes to climate science the lay public is hungry for knowledge, and many of them are eager, and well-prepared, for a level of sophistication and detail that can’t be found in lay journalism or even popular literature; An Inconvenient Truth isn’t enough. So blogs serve an incredibly useful purpose, enabling the interested and well-educated reader to share insights with researchers who are at the cutting edge of new knowledge.....................while blogs aren’t part of the machinery for legitimate scientific research, they’re an indispensible tool for communication and combating misinformation."

More to follow.

Posted by Maxine Clarke on April 10, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)

The first phase of the NIH review of the peer-review system was completed on 28 February , when the final draft of the 2007–2008 Peer Review Self-Study was submitted to NIH (US National Institutes of Health). In advance of NIH's announcement this Spring about the changes to be implemented, this month's (April) Editorial in Nature Medicine (14, 351; 2008) explores whether the NIH recommendations of a major overhaul of the system are likely to be cosmetic or curative.
The grant review process is onerous for both applicants and reviewers. According to the report, the success rate of first-time applicants (for A0 grants) is on the order of eight per cent. On average, investigators submit a grant three times before securing funding. Reviews can vary dramatically depending on the study section, and funding decisions often seem arbitrary. From a reviewer's standpoint, applications are long and numerous (about seven per reviewer), and the large time commitment dissuades many of the best-qualified people from participating in a process that lacks uniform review criteria, consistency and, arguably, objectivity. The report identifies seven main challenges to enhancing NIH peer review, each of which is described in the Nature Medicine Editorial. But, as the Editorial points out, the problem of too many researchers chasing too few dollars remains. More money won't alleviate administrative burden or improve the quality of reviews.
NIH's Enhancing peer-review report and next steps can be viewed here.

Posted by Maxine Clarke on April 07, 2008
Categories: Policy and Public | Permalink | Comments (0)

Nature Cell Biology continues its coverage of peer-review in its (free access) April Editorial 'Good review' (Nat. Cell Biol. 10, 371; 2008).
After stating the rationale for peer-review, the Editorial asks whether reviewing is essentially an altruistic activity, or whether there are significant rewards for doing it. "The biggest benefit of refereeing is probably to stay engaged with the scientific community in a sort of 'real-impact journal club'. We review less than a quarter of submitted manuscripts, with the aim of involving referees only for studies that seem worthy of publication. This saves time for authors, and referees know that the manuscripts they receive meet a minimum standard, so that refereeing should be an enriching experience. Nevertheless, the time commitment for refereeing is considerable and we are campaigning to ensure that this is taken into account in assessing research performance."
Although much has been said about alternatives and enhancements to 'single-blind' peer review, what actually constitutes a good referee report? The rest of the Editorial provides some answers, which are further explored at our author and peer-reviewers' website.
Nature Cell Biology's previous editorial (March 2008) is discussed here.

Posted by Maxine Clarke on April 04, 2008
Categories: Quality and value | Permalink | Comments (2)

Scientists know much more about their field than is ever published in peer-reviewed journals. Blogs can be a good medium with which to disseminate this tacit knowledge, according to Gavin Schmidt of the NASA Goddard Institute of Space Studies Columbia University, and co-founder of RealClimate.org, in a Commentary in the April issue of Nature Geoscience (1, 208; 2008). Dr Schmidt asks "why read a blog when you can go directly to the scientific literature? Unfortunately, access to new findings in the traditional way is harder than it should be. Many technical papers are behind pay-walls, which make it impractical and expensive for unaffiliated individuals to read them. More importantly, however, even when papers are freely available, they often do not provide the insight expected."
In another Commentary in the same issue of the journal (Nat. Geosci. 1, 209; 2008), Myles Allen of the University of Oxford agrees that "explaining science to journalists and the public on blogs is fast and efficient." But after his own bruising experience, he wonders: "is it all just too good to be true? Can science survive Web 2.0?"
After describing his own difficult experiences of having his own peer-reviewed work criticised and misinterpreted by people who would not subject their own conclusions to peer-review, Dr Allen concludes: "Everyone agrees we need to communicate science better to the general public. But more and faster should not be confused with better. I'm certainly not advocating closing science blogs or discouraging science websites. We just need to remember the basic courtesies that our doctoral supervisors took for granted: criticism of peer-reviewed results belongs in the peer-reviewed literature. Direct communication over the Internet, far from creating a level playing field, just ploughs it up and makes the game impossible. The problem is, without witty and cutting criticisms, what is the point of a blog? Sure, the peer-review system is creaking. Sure, science journalism sometimes trips up. But like Churchill's quip about democracy, it is the worst possible system for communicating scientific results, apart from all the alternatives that have been tried from time to time."
Yet Dr Schmidt remains optimistic: "Some may dismiss blogs as being a distraction from real scientific work, or of egging on the very controversies that we seek to diffuse. There is an element of truth to both of these claims. But the response should not be a return to the ivory tower. That simply leaves the field clear for those who prefer to confuse rather than enlighten. With the importance of science in policy decisions being more apparent than ever, our ability to do science and enhance its relevance in public life relies on the community's willingness to engage, inspire and inform. Blogs are one way to do that, and they can excel at providing the context that is so often missing in other media. Not every scientist needs to have one, but maybe every scientific field does."

Posted by Maxine Clarke on April 01, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)