Peer-to-peer

From Nature's Correspondence section this week (Nature, 447, 1052; 28 June 2007)
Sir -- I was excited when Nature launched its trial for open peer review last year, but disappointed by the outcome . I have also been tracking the progress of another open review journal, Biology Direct (see article in Nature's peer-review debate ). Even after a high-profile launch with guaranteed indexing by PubMed, this journal has published only 52 articles and received only two comments over 16 months.
In contrast to Biology Direct, another journal that offers open commenting, PLoS One , has published 1,189 articles in its first six months. But has PLoS One achieved its stated goal of post-publication open comments? I find that even the 'most annotated' category of articles usually receives just a few comments. The journal has recently replaced its 'most annotated' with a 'recently annotated' category. A check of all 'recently annotated' articles demonstrates that their commenting rates are low (zero or just a few), even for articles that are likely to have broad appeal and/or are in 'hot' research areas.
Why is there a general lack of interest among the scientific community in open commenting on submitted or published papers? I believe there are two main reasons. First, participation does not earn any tangible credit or benefit for the reviewers and commentators. Second, publicly critical comments are a risk for those who make them.
Shi V. Liu
Scientific Ethics, Apex, North Carolina

[Correction added by Maxine Clarke: Chris Surridge of PLOS One points out in a comment to the post that the journal published just over 550 articles in its first six months, not 1,189. Thank you, Chris.]

Posted by Maxine Clarke on June 29, 2007
Categories: Systems | Permalink | Comments (2)

Among the excellent content on Nature Reports Stem Cells is an especially exciting regular feature, "Inside the Paper", which zeros in on recent scientific papers in the field published in Nature and provides detailed expert comment and author responses drawn from the peer-review process. The editors intend this feature to let readers see the paper’s context, strengths and caveats, as well as make the peer- review process a little more transparent.

Two Inside the Paper features have been published this month. The first, Cloning from Chromosomes, discusses the Article by D. Egli et al., Developmental reprogramming after chromosome transfer into mitotic mouse zygotes, in Nature 447, 679–685 (2007). Read a panel of experts' comments on this interesting article, responses from the authors, and comment yourself on The Niche (the Nature Reports Stem Cells blog).
The second Inside the Paper (21 June), Rewriting in blood, is about a paper by I. M. Samokhvalov et al(Nature 446, 1056–1061; 2007), reporting a surprising origin for blood stem cells. As before, you can read some of the peer reviewer's comments, responses from the authors and invite you to add your contribution on The Niche.

Posted by Maxine Clarke on June 28, 2007
Categories: Quality and value, Systems | Permalink | Comments (0)

Nature Precedings is now out of "beta testing" and is launched. This new community service is described at Nautilus, the NPG blog for present and future authors. Submissions are screened by our professional curation team for relevance and quality, but are not subjected to peer review. High-quality contributions from biology, medicine (except clinical trials), chemistry and the Earth sciences are welcomed.

More details about Nature Precedings can be found here. Because they have not been peer-reviewed, many of the findings you read at Nature Precedings may be preliminary or speculative, and remain to be confirmed. Please bear this in mind when deciding how seriously to take them.

Submissions are not accepted from fields in the physical sciences that are are already well served by preprint servers such as arXiv.org. Content that considered to be non-scientific or pseudoscientific is rejected. We accept only genuine contributions from qualified scientists. This will usually require submitters to have a recognized academic affiliation. Incomplete submissions will also be rejected. This is a free service, so please help us to help you by completing all relevant sections of the submission form.

The reactions of scientists and publications to the launch of Nature Precedings can be found at this Connotea page, which is regularly updated. Here are one or two such articles:
Nature Precedings pre-print server for biomedical research
Nature Precedings: A nicer version of ArXiv[e] for biomedical research
Chemistry Central: A new preprint server from Nature.

Posted by Maxine Clarke on June 18, 2007
Categories: Open peer review, Systems, Technical solutions | Permalink | Comments (0)

This is the full text of an Editorial in today's issue of Nature (447, 754; 14 June 2007), on which we welcome your responses in the comments section to this post.

Everyone knows bad peer review when they come across it — but too few are nurturing good referees.

There is nothing more infuriating: you are an experienced scientist who has sent one of your best-ever papers to a journal, and what do you get back? A set of referees' comments that appals you. One reviewer asserts that the work is simply uninteresting and insufficiently original. Another displays wilful bias in relating their criticisms to results by a competitor whose outlook differs radically from yours. And a third has unreasonable expectations of what should be achieved. Not only are you upset, but your student co-author is devastated.

Continue reading "Mentors of tomorrow" »

Posted by Maxine Clarke on June 14, 2007
Categories: Mentoring and training | Permalink | Comments (4)

Steven Novella of the blog NeuroLogica writes an essay on peer review. " Peer-review is a critical part of the functioning of the scientific community, of quality control, and the self corrective nature of science", he writes. "But it is no panacea. It is helpful to understand what it is, and what it isn’t, its uses and abuses."
The essay is a short but informative account of the standard peer-review processes of many scientific journals, including a short paragraph about post-publication peer-review. At the time of writing this post, there are two comments on the subject of peer-review and fraud.
Dr Novella's "bottom line: peer-review is a necessary component of quality control in science, but is no guarantee of quality, and you have to know the details of the journal that is providing the peer-review."

Posted by Maxine Clarke on June 14, 2007
Categories: Systems | Permalink | Comments (0)

See Nature Reports Stem Cells for a fresh perspective on a paper reporting a technique for cloning from zygotes. In this fortnight’s Inside the Paper, a new form of scientific reporting pioneered by Nature Reports Stem Cells, you can read a moderated discussion between the authors and the paper's peer-reviewers. Readers can learn what the foremost experts in the field had to say about the submitted paper’s strengths and shortcomings. See what the authors saw, and read their responses as they revised their paper for eventual publication in Nature. You can also add your own comments on The Niche, the Nature Reports Stem Cells blog.
Featured paper: D. Egli et al. Developmental reprogramming after chromosome transfer into mitotic mouse zygotes. Nature 447, 679–685 (2007).

Posted by Maxine Clarke on June 13, 2007
Categories: Systems, Technical solutions | Permalink | Comments (0)

The Journal of Electronic Publishing

The Influence of Academic Values on Scholarly Publication and Communication Practices.
by Diane Harley, Sarah Earl-Novell, Jennifer Arter, Shannon Lawrence and C. Judson King
This paper was refereed by the Journal of Electronic Publishing 's peer reviewers.
From the abstract: This study reports on five disciplinary case studies that explore academic value systems as they influence publishing behavior and attitudes of University of California, Berkeley faculty. The case studies are based on direct interviews with relevant stakeholders — faculty, advancement reviewers, librarians, and editors — in five fields: chemical engineering, anthropology, law and economics, English-language literature, and biostatistics. The results of the study strongly confirm the vital role of peer review in the choices faculty make regarding their publishing behavior. The perceptions and realities of the reward system keep faculty strongly adhered to conventional, high-stature print publications (and their electronic surrogates) as the means of reporting research and having it institutionally evaluated. Perceptions of electronic-only publications are frequently negative because those venues are considered to lack strong peer review and are, consequently, believed to be of relatively lower quality. There is much more experimentation, however, with regard to means of in-progress communication, where single means of publication and communication are not fixed so deeply in values and tradition as they are for final, archival publication. We conclude that approaches that try to "move" faculty and deeply embedded value systems directly toward new forms of archival, "final" publication are destined largely to failure in the short-term. From our perspective, a more promising route is to (1) examine the needs of scholarly researchers for both final and in-progress communications, and (2) determine how those needs are likely to influence future scenarios in a range of disciplinary areas.
The full article can be seen at the journal's website.

Posted by Maxine Clarke on June 13, 2007
Categories: Quality and value | Permalink | Comments (0)

Picking up on the Nature Structural and Molecular Biology Editorial "New data at conferences, please", discussed here last week, Mario Pineda-Krch writes on his blog (Mario's Entangled Bank) about how the idea of the conference as a type of open peer-review process reminds him of the 1997 essay The academic Cathedral and the Bazaar by Eric S. Raymond.

In the Cathedral model the research process occurs behind closed doors among a exclusive group of researchers where progress is reported in peer-reviewed publications. In the Bazaar model the research is conducted in full view of the public......The basic tenet of the Bazaar model has in the field of software development termed Linus' law, i.e. "given enough eyeballs, all bugs are shallow". In an academic research environment the pros of the Bazaar model hinge on the fact that the more widely available the your research methodology and results is for public testing, scrutiny, and experimentation, the more rapidly all forms of errors and omissions will be discovered. In contrast, in the Cathedral model an inordinate amount of time and energy must be spent hunting for errors due to the limited number of eyeballs.

Mario, a postdoctoral researcher who works on ecological population dynamics at the University of California, Davis, believes that "academia would be a better place if more people would embrace this type of openness. Maybe the conferences setting could be the natural stage where the Bazaar movement could start."

Posted by Maxine Clarke on June 12, 2007
Categories: Systems | Permalink | Comments (0)

Inherently Responsive is the blog of the journal Heredity, published by NPG. As explained at the post Your chance to respond, the journal is seeking to improve its peer-review process using the opportunities provided by the web. One proposed model is to publish papers accompanied by referees’ comments, another is to publish correspondence columns in the journal after publication of the paper, to air legitimate differences of scientific opinion.
Inherently Responsive combines aspects of these two approaches. It publishes rapid feedback on papers that have appeared in Heredity: technical comments, relevant work published elsewhere and so on. Here is one example, about inheritance of litter size in Arctic foxes. It also plans to publish comments provided for public consumption by the peer-reviewers, so readers can see why the paper is controversial or why publication was recommended.
Inherently Responsive also features discussion on the editorial direction of the journal. In large part the content of Heredity is determined by what is submitted, but some of it is are commissioned. The blog seeks to obtain journal readers' views on whether important areas are being negelected.

Posted by Maxine Clarke on June 12, 2007
Categories: Systems | Permalink | Comments (0)

NIH Establishes Working Groups to Examine Peer Review

The Director of the US National Institutes of Health (NIH), Dr Elias A. Zerhouni, annnounced on Friday last week (8 June) the formation of two working groups to examine the NIH peer review process, with the goal of maximizing its effectiveness. The two groups will seek input from the scientific community, including investigators, scientific societies, academic institutions and health organizations, as well as from within NIH. The groups will study the context, criteria and culture of peer review to make sure the most talented individuals and reviewers are engaged in the process. More details, and the composition of the external and internal working groups, are available at the link at the top of this post.

Posted by Maxine Clarke on June 11, 2007
Categories: Systems | Permalink | Comments (0)

The Niche: How can journals improve peer review of cloning papers?

This week we welcome a new Nature blog, The Niche, the blog of the (also new) Nature Reports Stem Cells website. In one of its first posts, Nature editor Natalie De Witt writes:

In the aftermath of the Hwang scandal in 2006, Nature editors thought long and hard about whether journals could employ editorial procedures that might prevent publication of such fraudulent data in the future, at least in the area of cloning and nuclear transfer research. We queried several top scientists in the cloning and stem cell fields on this issue, and published the major conclusions in the editorial entitled "Standards for papers on cloning" Nature 439, 243 (2006).
Several of these scientists have agreed for us to publish abridged versions of their 2006 answers in The Niche. Open the Comments below to read the postings of George Daley, Shin-Ichi Nishikawa, Alan Trounson, Alan Colman, Robert Lanza, Teruhiko Wakayama, Bob Wall, and Mark Westhusin, on whether the Hwang scandal could have been prevented, and what tactics journals should implement in the future to tighten up cloning papers. Feel free to join in the discussion by posting your own comments.
The questions:
1. Is Nature's current review procedure adequate as it stands for refereeing of cloning papers?
2. If not, what improvements are needed?
3. Would the peer review process of cloning papers be improved were Nature to establish a checklist of minimal standard criteria for authors and referees to refer to during the peer review process?
4. If yes, what sort of experimental data should be considered and to what extent do the raw data need to be presented to reviewers, and to what extent should it be published?
5. In 2006, Nature asked for independent verification of the Hwang paper showing that Snuppy is a cloned dog. Do you feel this kind of independent verification should be requested for all cloning papers? If yes, what should it entail?

You can read all the responses here, and add your own comments.

Posted by Maxine Clarke on June 07, 2007
Categories: Quality and value | Permalink | Comments (0)

New data, please, is the plea in this month's Editorial in Nature Structural & Molecular Biology (14, 457; 2007). Some highlights:

The summer conference season is already in full swing. One of the great things about being a journal editor is the opportunity to go to different meetings (hopefully at beautiful locales), meet different people and learn about exciting new research. But this doesn't always happen — at least the part about seeing new research presented.
As editors, we are continually looking for the latest and most groundbreaking research, in the hope that we will be able to publish such work in our journal pages. As scientists, we are constantly looking to expand our knowledge into new areas and keep pace with the fields of research we are interested in. This is the main reason for us to attend meetings and conferences (and part of why we became editors in the first place).
-------
We understand that a lot of the hesitation about presenting a laboratory's most recent results may come from fear of being scooped. It is surprising, though, how much ownership comes with being the first to present a finding. In fact, the amount of discussion new results generate can be an indicator of how well received they will be when finally sent to peer review, since the question-and-answer session that follows the presentation can itself be viewed as an open review process. An advantage to this is that any problems or oversights may be caught ahead of time by the very same group of colleagues who will probably be involved in formal assessment of the results submitted for publication.
------
Whether it is in the form of a talk, a poster session or a high-profile publication, we are all here to learn and share what we know. It would be good if more scientists took advantage of the expertise gathered at the various meetings they attend by presenting their latest and greatest.

The complete text of the Editorial is available here (subscription or site licence required).

Posted by Maxine Clarke on June 06, 2007
Categories: Open peer review | Permalink | Comments (0)

FCW.com News - Data explosion strains peer review

The link above is a stimulating little article on FCW.com about how the US National Institute of Standards and Technology (NIST) deals with the overload of thermodynamic data reported in journals. From the FCW article:

"Recent improvements in measurement equipment mean that an already voluminous amount of thermodynamic data is doubling every 10 years.
That explosive growth is straining the traditional journal-based peer-review system and causing increasing numbers of errors to creep into the data. Companies in the chemical, pharmaceutical and energy industries depend on accurate data for their engineering applications and research projects.
As part of its responsibility for promoting U.S. competitiveness through standards and technology development, NIST worked with industry partners to create a standard data format and online system for verifying and disseminating thermodynamic data."

The XML-based system transforms the published data into a standard format and stores it in a central database that researchers can access via the Web. When authors submit data, the system automatically checks for inconsistencies and alerts the authors to any questionable data.

Posted by Maxine Clarke on June 04, 2007
Categories: Technical solutions | Permalink | Comments (0)

south by southwest festivals conferences
At the link above is a podcast of a session from the SXSW (South by South-West) Interactive Festival in Austin, Texas, held in March. From the conference website: "New publishing technologies challenge the traditional structure of peer-reviewed scientific journals. For hundreds of years the "article" has been the primary vehicle for conveying scientific information - but semantic markup, tagging and wiki are reconstructing scientific publications into a flexible and evolving concept." The panel looked at the social and legal implications of "Web 2.0" and the "Semantic Web" as they impact science and scientific knowledge. The moderator was John Wilbanks, Executive Director of Science Commons, Creative Commons, and one of the invited speakers was Timo Hannay, Director of Web Publishing of the Nature Publishing Group. The podcast has just been uploaded to the SXSW site, and is freely available by going to the link at the top of this post.

(Cross-posted on Nautilus.)

Posted by Maxine Clarke on June 04, 2007
Categories: Technical solutions | Permalink | Comments (0)