Grand unification or in miniature?

In his Nature Network summary of a recent publishing meeting, Yiorgos Apidianakis describes his opinion that a peer-review “score” is a more desirable (and efficient) indicator of scientific excellence than the currently used impact factor of the journal that publishes the work. Charles G. Jennings’s wrote, as part of Nature’s peer-review debate, “It is common to bemoan the over-reliance on quantitative markers such as impact factors for assessing scientists’ abilities (and indeed there is much to bemoan), but until committee members have time to read every paper on every applicant’s CV, they will have to rely at least in part on proxy indicators.” Dr Apidianakis believes that an ideal indicator would be a score from a unified peer-reviewing system, or a central agency “that will thoroughly, rigorously and objectively evaluate any given work to be published, using again specialized scientists as reviewers. Having an evaluation score from such an agency, scientists can include this score to their publication record.”

Dr Apidianakis thinks that such a system could work in practice, using the example of the NIH evaluation scoring system — while admitting it would be very expensive. Irrespective of this practical obstacle (as well as others) I wonder how the quality of the agency’s “score” can be standardized? The establishment of, in effect, one giant journal of research, with all the world’s scientists signed up to it as peer-reviewers, is a stimulating concept. But, given the many divergences of views within fields, how could a centralized scoring system work? The IPCC has attempted a similar kind of approach for one discipline, climate change — even though most scientists in the field broadly agree with the IPCC’s assessment of research output, this consensus requires massive bureaucratic baggage, including many international meetings and vast reports justifying decisions. Yet there is a substantial minority of scientific dissentors, and many members of the public are sceptical of a unified approach resulting in “science by endorsement”.

The current peer-review system works very well “in miniature”, whether as operated by those journals able to call on the most thoughtful scientists in a field who do the most cutting edge research, or by literature reviews or research ranking services, usually written or operated by one or a few individuals. Scientific research itself has benefited from large author collaborations across the whole spectrum of disciplines, from astronomy and nuclear physics to genomics and cell signalling, facing the challenge of making sense of vast quantities of data. Are there the same intellectual and innovative advantages to be gained by a single-managed peer-review system for all of science?

Peer Review and Scientific Consensus

Dr Robert Higgs of the Independent Institute, writes:

Journalists, politicians and advocacy groups refer to “peer-reviewed research” and “scientific consensus” as the authoritative last words on controversial matters involving the natural sciences, from climate change to stem-cell research and genetically engineered foods. But many people have an unrealistic view of how the scientific community actually works.

The peer-review process is not, contrary to popular belief, a nearly flawless system of Olympian scrutiny. Any editor of a peer-reviewed journal who desires to reject or accept a submission can easily do so by choosing appropriate referees.

Unfortunately, personal vendettas, ideological conflicts, professional jealousies, methodological disagreements, sheer self-promotion and irresponsibility are as much part of the scientific world as any other. Peer review cannot ensure that research is correct in its procedures and conclusions. A part of the work in every discipline – from the physical sciences to economics –consists of correcting previous mistakes.

At any given time, “scientific consensus” may exist about various matters. Over time, however, new interpretations, tests or observations may demolish that consensus. For instance, in the mid-1970s, an apparent scientific consensus existed that our planet was about to enter another Ice Age. Drastic proposals, such as exploding hydrogen bombs over polar icecaps to melt them. and damming the Bering Strait to prevent icy waters from entering the Pacific, were put forth by reputable scientists and seriously considered by the US government.

The truth is that scientific research at the upper echelons occurs within a fairly small world. Leading researchers attend the same conferences, belong to the same societies, review one another’s work for funding organizations, and so forth. If you do not belong to this tight fraternity, it becomes extremely difficult to gain a hearing for your work, to publish in a “top” journal, to acquire a government grant, to receive an invitation to participate in a scientific conference, or even to place your grad students in decent positions.

“Scientific consensus” often emerges because the members of this exclusive club, and those who support them, have too much invested in the reigning ideas to let go. In this context, it behooves bright young scientists not to rock the boat by challenging anything fundamental or dear to the hearts of those who constitute review committees of funders or journals. The terms “peer review” and “scientific consensus” often serve to suggest a process of disinterested neutrality and saintly pursuit of truth. Like every other human endeavour, however, science is conducted by people with the full range of human emotions and motives.

Good rules of thumb for the non-scientist might be the following: government-funded research that is used to justify that government’s policy should be suspect, whether or not it’s peer-reviewed; and the research of scientists who appear at press conferences in the company of politicians or activists whose agendas they are there to support should be suspect, whether or not the work upholds the consensus opinion.

Robert Higgs is Senior Fellow in Political Economy at the Independent Institute, editor of the quarterly journal The Independent Review, and the author of Depression War and Cold War, as well as numerous books and more than 100 articles in scholarly journals.

Post-publication review could aid skills and quality

Todd A. Gibson of the University of Colorado writes in Nature’s Correspondence (Nature 448, 408; 2007) pages:

Shi V. Liu’s correspondence, ‘Why are people reluctant to join in open review?’ (Nature 447, 1052; 2007; see also Peer to Peer discussion), struck a chord. Recently, I stayed my hand before submitting a comment to an online article, because the comment included as-yet unpublished research that I was reluctant to reveal in such a forum.

I believe that there are two ways to encourage online commenting. These would require little additional commitment, but would improve journal quality and enhance the development of review skills among young scientists.

First, journals could institute periodic post-publication review, in which the journal would solicit formal review of the article, focusing on how well its methods and results have held up, given the research that has been published in the intervening period. Such reviews would provide valuable historical perspective. Second, young scientists participating in journal clubs could be asked to derive and post a consensus comment on the article under discussion.

More on Nature Precedings and peer review

Mario of Mario’s Entangled Bank blog discusses an article in Wired magazine about Nature Precedings. Mario higlights Wired’s criticisms of Precedings for its “controversial” title, and for offering to upload scientists’ “effluvia and detritus” that would never appear in a journal. To the contrary, he writes, the site (on which he has been a beta tester) states that it “provides a rapid way to disseminate emerging results and new theories, solicit opinions, and record the provenance of ideas.” From Mario’s post:

“In other words, contrary to detritus and effluvia, Nature Precedings will host research at the cutting edge, some of which may have errors in it that will be “corrected” with the collaborative efforts of the research bazaar. Most of the posted research will, however, be correct (after all, it is your reputation that is at stake) and it’s early pre-print disemination will only be of benefit to the scientific community at large. This basically also addresses the last paragraph, “There’s also a danger of errors being missed that would have been picked up in the peer review process”. On the contrary, the recent PLoS retraction (see Show me the code and Exemplary retraction of high profile paper) confirms what we all academic researchers already know, that the peer review process is far from being fool proof. A forum like Nature Precedings, or arXiv for that matter, aid in the peer review process by disseminating pre-print cutting edge research findings thus allowing errors to be caught early on."

Why the reluctance for open peer review?

From Nature‘s Correspondence section this week (Nature, 447, 1052; 28 June 2007)

Sir — I was excited when Nature launched its trial for open peer review last year, but disappointed by the outcome . I have also been tracking the progress of another open review journal, Biology Direct (see article in Nature’s peer-review debate ). Even after a high-profile launch with guaranteed indexing by PubMed, this journal has published only 52 articles and received only two comments over 16 months.

In contrast to Biology Direct, another journal that offers open commenting, PLoS One , has published 1,189 articles in its first six months. But has PLoS One achieved its stated goal of post-publication open comments? I find that even the ‘most annotated’ category of articles usually receives just a few comments. The journal has recently replaced its ‘most annotated’ with a ‘recently annotated’ category. A check of all ‘recently annotated’ articles demonstrates that their commenting rates are low (zero or just a few), even for articles that are likely to have broad appeal and/or are in ‘hot’ research areas.

Why is there a general lack of interest among the scientific community in open commenting on submitted or published papers? I believe there are two main reasons. First, participation does not earn any tangible credit or benefit for the reviewers and commentators. Second, publicly critical comments are a risk for those who make them.

Shi V. Liu

Scientific Ethics, Apex, North Carolina

[Correction added by Maxine Clarke: Chris Surridge of PLOS One points out in a comment to the post that the journal published just over 550 articles in its first six months, not 1,189. Thank you, Chris.]

Inside the paper lifts the veil of peer review

Among the excellent content on Nature Reports Stem Cells is an especially exciting regular feature, “”https://blogs.nature.com/peer-to-peer/2007/06/inside_the_paper_postpublicati_1.html">Inside the Paper", which zeros in on recent scientific papers in the field published in Nature and provides detailed expert comment and author responses drawn from the peer-review process. The editors intend this feature to let readers see the paper’s context, strengths and caveats, as well as make the peer- review process a little more transparent.

Two Inside the Paper features have been published this month. The first, Cloning from Chromosomes, discusses the Article by D. Egli et al., Developmental reprogramming after chromosome transfer into mitotic mouse zygotes, in Nature 447, 679–685 (2007). Read a panel of experts’ comments on this interesting article, responses from the authors, and comment yourself on The Niche (the Nature Reports Stem Cells blog).

The second Inside the Paper (21 June), Rewriting in blood, is about a paper by I. M. Samokhvalov et al(Nature 446, 1056–1061; 2007), reporting a surprising origin for blood stem cells. As before, you can read some of the peer reviewer’s comments, responses from the authors and invite you to add your contribution on The Niche.

Nature Precedings is live

Nature Precedings is now out of “beta testing” and is launched. This new community service is described at Nautilus, the NPG blog for present and future authors. Submissions are screened by our professional curation team for relevance and quality, but are not subjected to peer review. High-quality contributions from biology, medicine (except clinical trials), chemistry and the Earth sciences are welcomed.

More details about Nature Precedings can be found here. Because they have not been peer-reviewed, many of the findings you read at Nature Precedings may be preliminary or speculative, and remain to be confirmed. Please bear this in mind when deciding how seriously to take them.

Submissions are not accepted from fields in the physical sciences that are are already well served by preprint servers such as arXiv.org. Content that considered to be non-scientific or pseudoscientific is rejected. We accept only genuine contributions from qualified scientists. This will usually require submitters to have a recognized academic affiliation. Incomplete submissions will also be rejected. This is a free service, so please help us to help you by completing all relevant sections of the submission form.

The reactions of scientists and publications to the launch of Nature Precedings can be found at this Connotea page, which is regularly updated. Here are one or two such articles:

Nature Precedings pre-print server for biomedical research

Nature Precedings: A nicer version of ArXiv[e] for biomedical research

Chemistry Central: A new preprint server from Nature.

A new form of post-publication peer-review

See Nature Reports Stem Cells for a fresh perspective on a paper reporting a technique for cloning from zygotes. In this fortnight’s Inside the Paper, a new form of scientific reporting pioneered by Nature Reports Stem Cells, you can read a moderated discussion between the authors and the paper’s peer-reviewers. Readers can learn what the foremost experts in the field had to say about the submitted paper’s strengths and shortcomings. See what the authors saw, and read their responses as they revised their paper for eventual publication in Nature. You can also add your own comments on The Niche, the Nature Reports Stem Cells blog.

Featured paper: D. Egli et al. Developmental reprogramming after chromosome transfer into mitotic mouse zygotes. Nature 447, 679–685 (2007).

The cathedral and the bazaar

Picking up on the Nature Structural and Molecular Biology Editorial “New data at conferences, please”, discussed here last week, Mario Pineda-Krch writes on his blog (Mario’s Entangled Bank) about how the idea of the conference as a type of open peer-review process reminds him of the 1997 essay The academic Cathedral and the Bazaar by Eric S. Raymond.

In the Cathedral model the research process occurs behind closed doors among a exclusive group of researchers where progress is reported in peer-reviewed publications. In the Bazaar model the research is conducted in full view of the public……The basic tenet of the Bazaar model has in the field of software development termed Linus’ law, i.e. “given enough eyeballs, all bugs are shallow”. In an academic research environment the pros of the Bazaar model hinge on the fact that the more widely available the your research methodology and results is for public testing, scrutiny, and experimentation, the more rapidly all forms of errors and omissions will be discovered. In contrast, in the Cathedral model an inordinate amount of time and energy must be spent hunting for errors due to the limited number of eyeballs.

Mario, a postdoctoral researcher who works on ecological population dynamics at the University of California, Davis, believes that “academia would be a better place if more people would embrace this type of openness. Maybe the conferences setting could be the natural stage where the Bazaar movement could start.”