Peer-to-peer

In a blaze of colour on the 11 November 'op-ed' (invited opinion) page of The New York Times, some scientists proclaimed that, based on analysis of brain-imaging data from just a handful of swing voters, they had divined what the rest of the undecided masses truly think about the upcoming US presidential elections. Apparently just asking them was simply not good enough.

So opens an Editorial in the current issue of Nature (450, 457; 2007), which goes on to describe how the authors of the New York Times piece used functional magnetic resonance imaging (fMRI) to scan the subjects' brains while looking at pictures of candidates. The Nature Editorial goes on:

"The op-ed work has not been published in a peer-reviewed journal, and the article is self-evidently too insubstantial in scientific detail to assess the strength of either the methods or the data. A group of cognitive neuroscientists was swift to object to its conclusions — which veer close to a modern-day phrenology — in a response to The New York Times.
The results described in the op-ed are apparently the claims of a commercial product posing as a scientific study. This is only partially transparent. Three of the authors list their affiliation with FKF Applied Research, a company based in Washington DC that is notorious for using similar brain-scan analysis to conclude which TV adverts aired during a major sporting event were most effective. In its own words, the company is a "business intelligence firm selling fMRI brain scan-based research to Fortune 500 companies".
Articles on The New York Times op-ed pages are opinionated by definition, and shouldn't normally require peer review. But here, the paper's editors have instead published the results of (to put it mildly) questionable scientific research, disseminating this information to millions of their readers who may not have the background to recognize for themselves the absurdity of some of the authors' conclusions.
Although it is a gross disservice to science and indeed to politics, it is a great deal for the company. Scientific publication would have required the authors to divulge their data and qualify their assumptions — and some journals might even have required that they declare their financial interests. Whatever the motives, seducing The New York Times' editors with the allure of Technicolor brains lighting up with Hillary Clinton angst yielded no more or less than a multimedia advertisement for the company's product to millions of readers."

Posted by Maxine Clarke on November 26, 2007
Categories: Ethics, Policy and Public, Quality and value | Permalink | Comments (0)

In an Editorial in today's Nature (450, 457-458; 2007), Replicator review, the journal describes its publication of "what we expect to be the final word on whether nuclear transfer can work in a primate — a paper by J. A. Byrne et al. showing not only that it is possible to clone primate embryos by somatic-cell nuclear transfer but also that precious embryonic stem cells can be derived from the embryos. If embryonic stem cells live up to their promise, the technology could be used to derive patient-tailored stem cells."
The Editorial goes on to describe how Nature took the unusual step of soliciting an independent verification of the paper during the process of peer review. "This is the first time that Nature has obtained second-party replication ahead of publication. It should not be seen as reflecting a mistrust of scientists in the cloning field or scepticism about this particular research group. Rather, our actions fulfilled a statement in an Editorial from 2006 that was conceived in the aftermath of the Hwang affair: "Keeping in mind the principle that extraordinary claims require extraordinary proof, Nature may in rare cases demand it."
Nature will continue to evaluate the need for such validation on a case-by-case basis. Meanwhile, researchers should consider maintaining tissue samples, and trying to establish and fulfil requirements that may go beyond the routine, as Byrne et al. did for their cloning paper. In that way, remarkable scientific developments can remain as free as possible from unwarranted speculation and controversy."

Also in this issue of the journal is a News and Views article on the new work, by Ian Wilmut and Jane Taylor (Nature 450, 485-486; 2007); a News story featuring an interview with one of the authors of the paper; and a Nature editors' one-paragraph summary of the work.

A blog discussion of peer-review challenges in stem-cell research and the peer-reviewers' reports for the paper by Byrne et al. can be found at Nature Reports Stem Cells.

Posted by Maxine Clarke on November 22, 2007
Categories: Quality and value | Permalink | Comments (0)

Via Action Potential:
The Neuroscience Peer-Review Consortium is an imaginative alliance of neuroscience journals whose goal is to "support efficient and thorough peer review of original research in neuroscience, speed the publication of research reports, and reduce the burden on peer reviewers." Any journal that is indexed in Medline and that publishes neuroscience research is invited to join. From January 2008, If a manuscript is not accepted by one journal in the consortium, the authors can submit their manuscript to a second consortium journal and have the reviews from the first journal forwarded to the second journal. This system is similar to the manuscript transfer system operated by the journals of Nature Publishing Group, but is subject- rather than publisher-specific.
A list of the journals that are participating so far can be seen here; further details about the consortium are here.
In a post discussing this initiative, Noah Gray of Action Potential (the Nature Neuroscience blog) explains that "many journals provide the opportunity not only to include a review of the manuscript for the authors, but also to communicate any additional feelings or opinions directly to the editors with confidence that the authors will not see these comments." To join the consortium, journals will have to stop using the "confidential comments to the editor" system. Noah asks his readers for their opinions as authors and reviewers of this proposal, and at time of writing this post, 30 people have replied.
One reviewer states: "I have consistently refused to place any information in the "confidential comments to the editors" review box. This is because I feel strongly that this means of communication is inherently suspect and open to abuse. In my opinion, every least communication between the editors and the reviewers should be available to the authors. Moreover, I feel that the authors must be given the right to respond to any criticism of their work. It is simply not fair to allow confidential comments to determine editorial decisions without this ability to respond."
Another reviewer, however, differs in finding a 'confidential comments' section "useful if you want to tell the editor that the paper is great, but would benefit greatly from a new analysis or new presentation style. In other words, if the author thinks you love the paper, they're less likely to agree to work on changes.
Thinking about it more, it seems as if confidential comments might be very useful in the event the referee knows about a conflict of interest, knows about conflicting data from the same lab that exists but is not in the manuscript, or knows of similar work in another lab. I've never had this experience, but I imagine it will happen sooner or later.
Why does the consortium plan insist on the absence of confidential comments? Presumably, if the author determines whether or not old reviews are to be considered, he will make that decision based on whether the reviews were good or bad. I don't see how he is helped or hurt by the exclusion of confidential comments."
There is a very thoughtful discussion at Action Potential, including some explanation from the consortium's working group, raising many pros and cons of confidential comments, to which we invite you to contribute.

Posted by Maxine Clarke on November 20, 2007
Categories: Ethics | Permalink | Comments (2)

There is quite a bit of advice and debate about peer review on various parts of nature.com at the moment, so I thought I'd round it up in this one post.
At The Sceptical Chymist blog, Catherine Goodman, an editor at Nature Chemical Biology, provides some thoughts on how to referee a paper, clarifying the journal's procedures and providing some tips.
Nature Medicine's August issue's Editorial set out what the journal seeks in its reviewers, asking readers how the journal can retain the best in the face of the plethora of requests to review from an increasing number of publications.
Over at Nature Network, there is a forum on peer-review in the Ask The Editor group, which addresses questions such as training of peer-reviewers and how Nature journal editors control the quality of the reviewing process.
In another Network Group, Nature Nanotechnology: Asia-Pacific and Beyond, journal editor Ai-Lin Chun asks "Do you know how to referee a paper?" and provides some answers to her readers. And the topic comes up again in the Publishing in the New Millennium forum, in the debate about the current state of science publishing.
As well as these blogs, forums and groups, the Nature Publishing Group Author and Reviewers' website has a page of peer-review information, including how to write a review for a Nature journal, what makes a good review, our peer-review policies, and why we operate the system of peer review that we do. And for a real, meaty discussion of how the Internet is affecting or may affect the publication process, you can read Nature's peer-review debate (22 articles) published in 2006.

Posted by Maxine Clarke on November 05, 2007
Categories: Journals | Permalink | Comments (0)

Bloggers for peer-reviewed research reporting, or BPR3, was conceived by scientists and others who write informally about research on the Internet as a simple way to denote that a blog post or website article is discussing peer-reviewed work. Their mission statement: "Bloggers for Peer-Reviewed Research Reporting strives to identify serious academic blog posts about peer-reviewed research by offering an icon and an aggregation site where others can look to find the best academic blogging on the Net."
As well as creating an icon for bloggers to denote when their posts concern peer-reviewed research, the organization will host a central web site where snippets from these posts will be displayed, with links back to the original posts. Readers will be able to choose topics of interest and view only those posts, if they wish.
Eventually, the intention is that bloggers will be able to enter a DOI (digital object identifier) or other unique identifier, and automatically generate code to post the icon, link to the post to the BPR3 site and its aggregation tools, and generate a properly formatted research citation which links to the original article.
Now, the first stage is complete -- the icons have been created (with a lot of helpful input from large numbers of science bloggers). They must be popular, as the site from which people can collect their code has been down for a couple of days due to excessive traffic, presumably. However, it is now up and viewable, so if you want to pick up an icon for your own blog, or just find out more details of the project, please visit this posting. From this post (29 October): "Anyone can use these icons to show when they're making a serious post about peer-reviewed research, rather than just linking to a news article or press release. Within a month, these blog posts will also be aggregated here, so everyone can go to one place to locate the most serious, thoughtful analysis and commentary on the web."
Inevitably, given the engagingly self-referential and gossipy nature of blogging, release of these icons attracted a lot of comment and discussion in the blogosphere. Dave Munger, the driving force behind BPR3, has collected links to all these articles into one post here.
Perhaps in some of this discussion, this question has been addressed already, but as I am afraid I do not have time to read all of the 26 (so far) articles, I raise it here. The icon seems to me an excellent way to indicate that the subject of a blog post is a peer-reviewed research article (so long as there is a mechanism to report abuse and remove the icon from posts that use it incorrectly). And the intentions of BPR3 in providing links to the original research article being discussed are admirable. However, these indicators in themselves do not seem to me necessarily to be an indicator of quality of the blog post itself. On a blog, anyone can write anything about anything, whether or not the topic under discussion is peer-reviewed. A lot of traffic to, or comments on, a blog post is not in itself an indicator of quality. (It could, indeed, be the opposite.)
After the blog posts are themselves linked to the original research article, it will be possible for them to be formally cited, making them eligible for aggregation into citation databases. Blog posts that enter the mainstream scientific debate in this sense would have a quality indicator associated with the post itself rather than the research they describe. Is this the intention?

Posted by Maxine Clarke on November 02, 2007
Categories: Perspectives, Quality and value | Permalink | Comments (4)