Peer-to-peer

A News story in the 12 June issue of Nature (453, 823; 2008) by Meredith Wadman:
Responding to hundreds of critical comments, the US National Institutes of Health (NIH) has reversed several controversial proposals made in February as part of a year-long effort to overhaul the agency's peer-review system (see Nature 451, 1035; 2008).
As part of an initiative called Enhancing Peer Review, announced in a finalized form on 6 June, the agency will spend at least $200 million annually over the next five years to foster groundbreaking, investigator-initiated research. Of that, at least $250 million will go to a new beast: a Transformative R01 Award, a reach-for-the-skies version of the NIH's basic grant. The remaining $750 million will go to existing awards that reward risk and innovation: the Eureka, New Innovator and Pioneer awards.
The changes “are concrete solutions that will maximize flexibility, remove any unnecessary burden, stimulate new innovation and promote transformative research”, says NIH director Elias Zerhouni.
They include rewards for long-serving reviewers; a streamlined, 12-page R01 grant application, down from 25; and a seven-point, integer scoring scale for grant applications, which will be assessed across five criteria: impact, investigators, innovation, feasibility and environment. Current applications are graded on a 41 point scale, from 1.0 to 5.0, raising complaints that they claim a degree of accuracy that can't be scientifically defended.
Among the controversial proposals shelved by the agency was a recommendation that all applications, even those on a second or third submission, would be treated as new, without reviewer access to prior reviews.
Gone, too, is the category “not recommended for resubmission”, which had been suggested for dismal applications. Scientists felt that branding projects with “a clear, checkbox-driven stigma is bad, that it could have unintended consequences”, Jeremy Berg, director of the National Institute of General Medical Sciences, told the advisory committee.
Berg and Lawrence Tabak, director of the NIH's dental institute, head the group that developed the recommendations and are charged with implementing them over the next 18 months.
The agency also jettisoned a “minimum effort requirement” that would have required principal investigators to commit at least 20% of their time to any single NIH grant — an item of particular concern for 'grandee grantees' (see Nature 452, 258–259; 2008). Instead, grantees will need to indicate if they will have more than $1 million in cumulative NIH funding.

Posted by Maxine Clarke on June 12, 2008
Categories: Grants, tenure and positions, Systems | Permalink | Comments (1)

The Editorial 'Working double-blind' (Nature 451, 605–606; 2008), also republished on this blog and stimulating more than 70 comments, referred to a study (1) that found more female first-author papers were published using a double-blind, rather than a single-blind, peer-review system. The data reported in ref. 1 have now been re-examined (2). The conclusion of ref. 1, that Behavioral Ecology published more papers with female first authors after switching to a double-blind peer-review system, is not in dispute. However, ref. 2 reports that other similar ecology journals that have single-blind peer-review systems also increased in female first-author papers over the same time period. After re-examining the analyses, Nature has concluded that ref. 1 can no longer be said to offer compelling evidence of a role for gender bias in single-blind peer review. In addition, upon closer examination of the papers listed in PubMed on gender bias and peer review, we cannot find other strong studies that support this claim. Thus, we no longer stand by the statement in the fourth paragraph of the Editorial, that double-blind peer review reduces bias against authors with female first names.
References
1. Budden, A. E. et al . Trends Ecol. Evol. 23, 4–6 (2008).
2. Webb, T. J. , O'Hara, B. & Freckleton, R. P. Trends Ecol. Evol. doi:10.1016/j.tree.2008.03.003 (2008).

Posted by Maxine Clarke on June 05, 2008
Categories: Ethics, Quality and value, Systems | Permalink | Comments (2)

In the latest Nature Reports Stem Cells Inside the Paper feature, senior Nature editor Natalie DeWitt discusses the paper by H. H. Chang et al. 'Transcriptome-wide noise controls lineage choice in mammalian progenitor cells', published in Nature 453, 544-547 (2008).

The Editor's summary of this paper: Even in clonal populations of cells, there is significant phenotypic variation from cell to cell. This could reflect the 'noise' inherent in gene expression: or the various cell states could represent stable phenotypic variants. Chang et al. analysed the behaviour of an 'outlier' in clonal populations of mouse haematoipoietic stem cells that had very high expressions of the stem cell marker Sca-1 and found that outliers possessed distinct transcriptomes. Though the transcriptomes eventually reverted back to that of the median cells, while they differed they could drive the cells to express characteristics of distinct cell fates. Thus clonal heterogeneity of gene expression may not be due to noise in the expression of individual genes, but rather is a manifestation of metastable states of a slowly fluctuating transcriptome. These fluctuations may govern the reversible, stochastic priming of multipotent progenitor cells in cell fate decision.

The Nature Reports Stem Cells article provides the details of what the peer-reviewers thought of this paper when it was submitted, and how the authors responded. The initial view of the three peer-reviewers can be summarized as follows:

Reviewer 1 In summary, we believe that this paper reports a novel and important biological mechanism for differentiation: it gives evidence that slow stochastic variations in stem cell state last for a long time and result in different fates. It relates the recent finding of slow fluctuations in human protein levels to the biological outcome of cell fate, and finds long lasting differences in transcriptomes in different subpopulations. It is an excellent choice for Nature, provided that the comments are addressed.

Reviewer 2 Although the phenomenon described is immensely interesting and the idea of heterogeneity being retained within even clonal populations of cells is plausible, the authors merely describe this phenomenon and in some instances, do not provide conclusive data to support their interpretation. If a mechanism was determined this would definitely aid in its novelty and interest. In its current form, I believe this manuscript should not be accepted at Nature.

Reviewer 3 This is a very timely and interesting paper, that should be of interest to a broad range of researchers. However, I have some serious concerns over the modeling.

The details of the reviewers' concerns, and the authors' response, can be read at Nature Reports Stem Cells,

You can also read a set of question-and-answer sessions between editor Monya Baker and several of the peer-reviewers of the recent Nature Insight on Regenerative Medicine, here on The Niche, the blog of Nature Reports Stem Cells.

Posted by Maxine Clarke on June 02, 2008
Categories: Systems | Permalink | Comments (0)

Nature Neuroscience is joining a consortium of journals that enables reviews to be transferred from one journal to another, while allowing authors, referees and editors to control their degree of participation in the system flexibly. The reasons for the decision are explained in this month's (April) Editorial (Nat. Neurosci. 11, 375; 2008). Briefly, in January, a group of editors, supported by the Society for Neuroscience, implemented a system for transfer of submitted manuscripts between journals that allows voluntary participation by authors, referees and editors, known as the Neuroscience Peer Review Consortium. This consortium reduces the overall reviewing workload of the community by allowing authors to continue the initial review process when their paper moves from one consortium journal to another, once the paper has been rejected or withdrawn from the first journal. This arrangement is similar to the manuscript transfer system that has been available within the Nature family of journals , and all the other journals published by Nature Publishing Group, for almost a decade.
The neuroscience transfer system, described in detail in the Editorial and at the Neuroscience Peer Review Consortium website, is voluntary for authors and peer-reviewers. Journal editors have full discretion in deciding how to use transferred reviews. The receiving editor may choose to accept or reject a paper based on these reviews, without further consideration; to send the paper to some or all of the previous referees for evaluation of the authors' revisions; or to request a fresh set of reviews from new referees. Only comments to the authors are transferred to the receiving journal. Confidential comments to the editors are not passed along. Thus, to ensure transparency in the review process, both at Nature Neuroscience and at other journals after the paper has been transferred, journal editors encourage referees to include all their concerns about the paper in comments to the authors. According to the Nature Neuroscience Editorial, the small amount of extra time required to phrase comments diplomatically for the authors should be more than counterbalanced by the resulting improvement in the peer-review process.
Many members of the community have strong views on the issue of confidential comments, which can be found on the Action Potential blog, and which has previously been discussed at Peer to Peer. The success of the neuroscience journals' transfer system will be evaluated at the end of this year. Nature Neuroscience editors will also be evaluating the journal's participation on an ongoing basis, so they encourage authors, referees and readers to share their comments with the editors, either on Action Potential blog or privately by e-mail.

Posted by Maxine Clarke on March 31, 2008
Categories: Ethics, Quality and value, Systems | Permalink | Comments (0)

The newly launched journal Scholarly Research Exchange, published by Hindawi, is for original research articles in all areas of science, technology and medicine. The journal is operating a transparent peer-review system, in which authors and reviewers interact directly throughout the peer-review process. Authors submitting to Scholarly Research Exchange suggest potential reviewers, who are then approved by the journal's editors. Reviewers are asked to provide an assessment of the quality of the manuscript, a written critique for the authors, and a written commentary for the journal's readers. When the journal accepts its first articles for publication, the reviewers' commentaries and their assessment of the manuscript's quality will be published with them. In addition to these reviewers' evaluations, members of the scientific community will be able to participate in a discussion around every manuscript.
A journal with a similar, but not identical, peer-review process is Biology Direct, published by BioMed Central, currently covering the areas of genomics, bioinformatics and systems biology, immunology, and mathematical biology. Biology Direct makes the author responsible for obtaining reviewers' reports, via the journal's editorial board, operates an open peer-review process, and publishes reviewers' reports with the articles. It has been publishing articles since January 2006.

Posted by Maxine Clarke on March 12, 2008
Categories: Systems | Permalink | Comments (0)

How many papers does the typical researcher review in a year? How long do they take? And why do they do it? For the answers, read the Editorial in this month's (March) issue of Nature Nanotechnology, "Who'd be a referee?" (3, 119; 2008). The Editorial reports some of the findings of the recent Publishing Research Consortium report on peer-review (previously discussed at Peer to Peer), whose survey revealed that the "average review takes about 8.6 hours (with a median of about 5 hours) and is completed within 3–4 weeks, although there are significant differences between the four broad subject areas covered by the survey: in the physical sciences and engineering, for instance, the average (mean) is 10.4 hours, compared with 6.3 for clinical researchers. Of course, on top of this, many researchers spend considerable time reviewing grant applications for funding agencies, which can be equally onerous and possibly even more important than reviewing papers."
For similar statistics, and a more general discussion of the peer-review process in publishing, see the Nature Nanotechnology Editorial, or see here for Nature Cell Biology's take.

Posted by Maxine Clarke on March 10, 2008
Categories: Quality and value, Systems | Permalink | Comments (0)

This month's (March 2008) free-access Editorial in Nature Cell Biology (10, 247; 2008) addresses the journal's peer-review process: specifically, what the journal does to ensure that its selection process is fair.
From the Editorial:
"A legitimate question for editors at Nature Cell Biology is whether we are considering alternatives to the status quo of anonymous peer review, in particular, why we do not use a 'double blind' process (which received significant endorsement in a recent survey of the Publishing Research Consortium). The existing process, based on a thorough pre-selection by five full-time editors and subsequent external peer review by carefully selected referees, works well — individual stories of woe notwithstanding."
The editorial goes on to outline how changes to the system, when being considered, must show a demonstrable improvement to the process. Several of the commonly proposed alternatives to the 'single blind' system are discussed in this light (see here for a recent popular debate on the topic at this blog). The Editorial concludes with a summary of the process as currently run by the Nature Cell Biology editors, together with the journal's planned enhancments.
The editors welcome your views on the Editorial as comments to this post.
Further information about the Nature journals' peer-review policies are available at the authors' and reviewers' website.
Connotea tags for peer-review.

Posted by Maxine Clarke on March 06, 2008
Categories: Journals, Systems | Permalink | Comments (0)

This is the text of an editorial published in Nature in the 7 February issue (Nature 451; 605-606; 2008) . We, the journal editors, welcome your comments and suggestions.
Corrected 4 June 2008: Nature 453, 711 (2008).
Please see end of this post for text of correction correction to this editorial*
Should there be author anonymity in peer review?

Double-blind peer review, in which both authors and referees are anonymous, is apparently much revered, if not much practised. The Publishing Research Consortium (PRC) has assessed attitudes towards peer review among 3,000 academics in an international survey across the sciences and humanities. The results, released last month, strongly affirm the value of peer review [See earlier posting on Peer to Peer.]. They also highlight that 71% have confidence in double-blind peer review and that 56% prefer it to other forms of review. Support is highest with those who have experienced it (the humanities and social sciences) or where it is perceived to do the most good (among female authors). The least enthusiastic group is editors. So is it time for editors, and those at Nature in particular, to reconsider their position?
If referees know the authors' identities, it may leave the latter vulnerable to biases about them or their previous work, their gender, their nationality or their being new to an area of research. But the PRC survey supports the contention of Nature and others that identifying authors stimulates referees to ask appropriate questions (for example, differentiating between a muddy technical explanation and poor experimental technique). Knowing author identities also makes it easier to compare the new manuscript with the authors' previously published work, to ensure that a true advance is being reported. And knowing rather than guessing the identities of authors encourages reviewers to raise potential conflicts of interest to the editors.
Is there evidence that double-blind peer review presents a better alternative? It would do so if it generated more constructive comments in the minds of editors and authors, or if the identity of authors were truly protected, or if biases were reduced. So far, the jury is out. Although at least one study in the biomedical literature has suggested that double-blind peer review increases the quality of reviews, a larger study of seven medical journals (refs 2, 3) indicated that neither authors nor editors found significant difference in the quality of comments when both referees and authors were blinded. Referees could identify at least one of the authors on about 40% of the papers, undermining the raison d'être for double-blinding. The editors at the Public Library of Science abandoned double-blind peer review because too few requested it and authors were too readily identified.
The one bright light in favour of double-blind peer review is the measured reduction in bias against authors with female first names (shown in numerous studies, such as ref. 4 [previously discussed on Peer to Peer]). This suggests that authors submitting papers to traditionally minded journals should include the given names of authors only on the final, published version.
The double-blind approach is predicated on a culture in which manuscripts-in-progress are kept secret. This is true for the most part in the life sciences. But some physical sciences, such as high-energy physics, share preprints extensively through arXiv, an online repository. Thus, double-blind peer review is at odds with another 'force for good' in the academic world: the open sharing of information. The PRC survey found that highly competitive fields (such as neuroscience) or those with larger commercial or applied interests (such as materials science and chemical engineering) were the most enthusiastic about double-blinding, whereas fields with more of a tradition for openness (astronomy and mathematics) were decidedly less supportive.
Where does this leave journals? Editors have the responsibility to provide a neutral bridge between referees and authors and so may help to better shield authors from bias. Easily said! The evidence of the PRC survey suggests little faith in that impartiality, but editors — certainly at Nature and its related journals — take that responsibility seriously.
Nature's policies over the years have generally moved towards greater transparency. Coupling that with the lack of evidence that double-anonymity is beneficial makes this journal resistant to adopting it as the default refereeing policy any time soon. But many of our readers are referees as well as authors. We welcome their views on author anonymity from both vantage points.
1. Publishing Research Consortium Peer Review in Scholarly Journals (Mark Ware Consulting, Bristol, 2008).
2. Justice, A. C. et al. J. Am. Med. Assoc. 280, 240–242 (1998).
3. Cho, M. K. et al. J. Am. Med. Assoc. 280, 243–245 (1998).
4. Budden, A. E. et al. Trends Ecol. Evol. 23, 4–6 (2008).
The Nature journals' peer-review policies are described at our Author and Reviewers' website, where you can also find links to our editorials on the topic, all free to access.

CORRECTION to the editorial:
The Editorial 'Working double-blind' (Nature 451, 605–606; 2008) referred to a study1 that found more female first-author papers were published using a double-blind, rather than a single-blind, peer-review system. The data reported in ref. 1 have now been re-examined2. The conclusion of ref. 1, that Behavioral Ecology published more papers with female first authors after switching to a double-blind peer-review system, is not in dispute. However, ref. 2 reports that other similar ecology journals that have single-blind peer-review systems also increased in female first-author papers over the same time period. After re-examining the analyses, Nature has concluded that ref. 1 can no longer be said to offer compelling evidence of a role for gender bias in single-blind peer review. In addition, upon closer examination of the papers listed in PubMed on gender bias and peer review, we cannot find other strong studies that support this claim. Thus, we no longer stand by the statement in the fourth paragraph of the Editorial, that double-blind peer review reduces bias against authors with female first names.
References
Budden, A. E. et al . Trends Ecol. Evol. 23, 4–6 (2008).
Webb, T. J. , O'Hara, B. & Freckleton, R. P. Trends Ecol. Evol. doi:10.1016/j.tree.2008.03.003 (2008).

Posted by Maxine Clarke on February 06, 2008
Categories: Systems | Permalink | Comments (81)

Much attention is being given to a Commentary in the current issue of Nature (Nature 451, 379-399; 2008), A tale of two citations, some of which I have attempted to encapsulate in this Nautilus post, for those interested. Although the issues immediately concern a possible increase in duplicate publication and plagiarism, as detected by software systems and database searches, peer reviewers are an integral part of the check/balance procedures that journals use. For this reason, I thought it well worth highlighting here the comment by Brian Derby at the Nature Network forum currently discussing these questions. Part of Dr Derby's response:

"As a referee I have identified duplicate or severely overlapping content while reviewing papers in the past (for reasonably high profile/impact factor journals). I do not search for duplication routinely but, as someone who is used to referee papers in particular niche areas, I received both papers in one instance and in another I had read an on-line pre-pub before receiving the duplicate. The authors will not be named as that would break referee confidentiality but they were from well known institutions in the developed world.
What was the common factor (apart from the paper!) – the authors were relatively junior new appointments. Younger academics seem to feel themselves under a lot of pressure to publish. In my department I believe that my younger colleagues are much more sensitive to impact factor than is possibly healthy when they consider where to publish an article."

Posted by Maxine Clarke on January 28, 2008
Categories: Ethics, Quality and value, Systems | Permalink | Comments (1)

The Publishing Research Consortium publishes a study this month (January 2008) in whch more than 3,000 senior authors, reviewers and editors were asked about the peer-review system. The conclusions are that researchers want to "improve, not change, the system of peer review for journal articles". According to the report, a summary of which is available (1.7 MB; PDF), more than 93 per cent of respondents believe that peer review is necessary, and more than 85 per cent say that it helps to improve scientific communications and increases the overall quality of published papers.
Although many respondents pointed out the operational difficulties in double-blind peer review, two-thirds of respondents felt that it is the most objectively fair system, compared with single-blind (the current prevalent system). Alternatives such as post-publication and open peer-review were not popular.
While of the majority of respondents saw peer review as an effective filter for research, some did not think it was effective at detecting plagiarism, fraud or misconduct. Interestingly, most reviewers among the respondents thought that paying peer-reviewers would be too expensive for publishers; most of them said that they perform reviewing as part of their support to their research community.
The full report is available here (1 GB; PDF). According to the Publishing Research Consortium, the main objective of the study was "to measure the attitudes and behaviour of the academic community with regard to peer review. This will inform debate concerning peer review, and underpin discussions, either in discussion lists or at future workshops/conferences."
This new report comes as the NIH (National Institutes of Health) finish analysing the thousands of responses to their assessment of grant peer review. Lawrence Tabak and colleagues are filtering the list into a set of key recommendations, which will be given to Elias Zerhouni, director of NIH, at the end of February.
Update, 29 Jan 2008. Nature Neuroscience discusses the NIH peer-review exercise in its February issue Editorial (Rethinking grant review Nature Neuroscience 11, 119; 2008).

Posted by Maxine Clarke on January 25, 2008
Categories: Quality and value, Systems | Permalink | Comments (0)

Double-blind peer review, in which neither author nor reviewer identity are revealed, was introduced by the journal Behavioral Ecology in 2001. Amber E. Budden et al., in an article published in Trends in Ecology and Evolution this month (Trends Ecol. Evol. 23, 4-6; 2008) report "a significant increase in female first-authored papers" compared with a similar journal, Behavioral Ecology and Sociobiology. From the authors' conclusions:
"A difference of 7.9% in the proportion of female first-authored papers following the implementation of double-blind review in BE is three times greater than the recorded increase in female ecology graduates in the USA across the same time period and represents a 33% increase in the representation of female authors. Furthermore, this increased representation of female authors more accurately reflects the (US) life sciences academic workforce composition, which is 37% female.
The consequences of this shift could extend beyond publications. If females are less successful in publishing research on account of their gender, then given the current practices associated with appointment and tenure, and the need for women dramatically to out-compete their male counterparts to be perceived as equal [C. Wenneras and A. Wold, Nepotism and sexism in peer-review, Nature 387 341–343; 1997] any such publication bias impedes the progress of women to more advanced professional stages."

Posted by Maxine Clarke on January 21, 2008
Categories: Ethics, Systems | Permalink | Comments (0)

In his Nature Network summary of a recent publishing meeting, Yiorgos Apidianakis describes his opinion that a peer-review "score" is a more desirable (and efficient) indicator of scientific excellence than the currently used impact factor of the journal that publishes the work. Charles G. Jennings's wrote, as part of Nature's peer-review debate, “It is common to bemoan the over-reliance on quantitative markers such as impact factors for assessing scientists’ abilities (and indeed there is much to bemoan), but until committee members have time to read every paper on every applicant’s CV, they will have to rely at least in part on proxy indicators.” Dr Apidianakis believes that an ideal indicator would be a score from a unified peer-reviewing system, or a central agency "that will thoroughly, rigorously and objectively evaluate any given work to be published, using again specialized scientists as reviewers. Having an evaluation score from such an agency, scientists can include this score to their publication record."

Dr Apidianakis thinks that such a system could work in practice, using the example of the NIH evaluation scoring system -- while admitting it would be very expensive. Irrespective of this practical obstacle (as well as others) I wonder how the quality of the agency's "score" can be standardized? The establishment of, in effect, one giant journal of research, with all the world's scientists signed up to it as peer-reviewers, is a stimulating concept. But, given the many divergences of views within fields, how could a centralized scoring system work? The IPCC has attempted a similar kind of approach for one discipline, climate change -- even though most scientists in the field broadly agree with the IPCC's assessment of research output, this consensus requires massive bureaucratic baggage, including many international meetings and vast reports justifying decisions. Yet there is a substantial minority of scientific dissentors, and many members of the public are sceptical of a unified approach resulting in "science by endorsement".

The current peer-review system works very well "in miniature", whether as operated by those journals able to call on the most thoughtful scientists in a field who do the most cutting edge research, or by literature reviews or research ranking services, usually written or operated by one or a few individuals. Scientific research itself has benefited from large author collaborations across the whole spectrum of disciplines, from astronomy and nuclear physics to genomics and cell signalling, facing the challenge of making sense of vast quantities of data. Are there the same intellectual and innovative advantages to be gained by a single-managed peer-review system for all of science?

Posted by Maxine Clarke on December 04, 2007
Categories: Quality and value, Systems | Permalink | Comments (0)

Dr Robert Higgs of the Independent Institute, writes:

Journalists, politicians and advocacy groups refer to “peer-reviewed research” and “scientific consensus” as the authoritative last words on controversial matters involving the natural sciences, from climate change to stem-cell research and genetically engineered foods. But many people have an unrealistic view of how the scientific community actually works.
The peer-review process is not, contrary to popular belief, a nearly flawless system of Olympian scrutiny. Any editor of a peer-reviewed journal who desires to reject or accept a submission can easily do so by choosing appropriate referees.
Unfortunately, personal vendettas, ideological conflicts, professional jealousies, methodological disagreements, sheer self-promotion and irresponsibility are as much part of the scientific world as any other. Peer review cannot ensure that research is correct in its procedures and conclusions. A part of the work in every discipline – from the physical sciences to economics –consists of correcting previous mistakes.
At any given time, “scientific consensus” may exist about various matters. Over time, however, new interpretations, tests or observations may demolish that consensus. For instance, in the mid-1970s, an apparent scientific consensus existed that our planet was about to enter another Ice Age. Drastic proposals, such as exploding hydrogen bombs over polar icecaps to melt them. and damming the Bering Strait to prevent icy waters from entering the Pacific, were put forth by reputable scientists and seriously considered by the US government.
The truth is that scientific research at the upper echelons occurs within a fairly small world. Leading researchers attend the same conferences, belong to the same societies, review one another’s work for funding organizations, and so forth. If you do not belong to this tight fraternity, it becomes extremely difficult to gain a hearing for your work, to publish in a “top” journal, to acquire a government grant, to receive an invitation to participate in a scientific conference, or even to place your grad students in decent positions.
“Scientific consensus” often emerges because the members of this exclusive club, and those who support them, have too much invested in the reigning ideas to let go. In this context, it behooves bright young scientists not to rock the boat by challenging anything fundamental or dear to the hearts of those who constitute review committees of funders or journals. The terms "peer review" and "scientific consensus" often serve to suggest a process of disinterested neutrality and saintly pursuit of truth. Like every other human endeavour, however, science is conducted by people with the full range of human emotions and motives.
Good rules of thumb for the non-scientist might be the following: government-funded research that is used to justify that government’s policy should be suspect, whether or not it’s peer-reviewed; and the research of scientists who appear at press conferences in the company of politicians or activists whose agendas they are there to support should be suspect, whether or not the work upholds the consensus opinion.

Robert Higgs is Senior Fellow in Political Economy at the Independent Institute, editor of the quarterly journal The Independent Review, and the author of Depression War and Cold War, as well as numerous books and more than 100 articles in scholarly journals.

Posted by Maxine Clarke on September 17, 2007
Categories: Quality and value, Systems | Permalink | Comments (5)

Todd A. Gibson of the University of Colorado writes in Nature's Correspondence (Nature 448, 408; 2007) pages:

Shi V. Liu's correspondence, 'Why are people reluctant to join in open review?' (Nature 447, 1052; 2007; see also Peer to Peer discussion), struck a chord. Recently, I stayed my hand before submitting a comment to an online article, because the comment included as-yet unpublished research that I was reluctant to reveal in such a forum.
I believe that there are two ways to encourage online commenting. These would require little additional commitment, but would improve journal quality and enhance the development of review skills among young scientists.
First, journals could institute periodic post-publication review, in which the journal would solicit formal review of the article, focusing on how well its methods and results have held up, given the research that has been published in the intervening period. Such reviews would provide valuable historical perspective. Second, young scientists participating in journal clubs could be asked to derive and post a consensus comment on the article under discussion.

Posted by Maxine Clarke on August 09, 2007
Categories: Mentoring and training, Systems | Permalink | Comments (0)

Mario of Mario's Entangled Bank blog discusses an article in Wired magazine about Nature Precedings. Mario higlights Wired's criticisms of Precedings for its "controversial" title, and for offering to upload scientists' "effluvia and detritus" that would never appear in a journal. To the contrary, he writes, the site (on which he has been a beta tester) states that it "provides a rapid way to disseminate emerging results and new theories, solicit opinions, and record the provenance of ideas." From Mario's post:

"In other words, contrary to detritus and effluvia, Nature Precedings will host research at the cutting edge, some of which may have errors in it that will be "corrected" with the collaborative efforts of the research bazaar. Most of the posted research will, however, be correct (after all, it is your reputation that is at stake) and it's early pre-print disemination will only be of benefit to the scientific community at large. This basically also addresses the last paragraph, "There's also a danger of errors being missed that would have been picked up in the peer review process". On the contrary, the recent PLoS retraction (see Show me the code and Exemplary retraction of high profile paper) confirms what we all academic researchers already know, that the peer review process is far from being fool proof. A forum like Nature Precedings, or arXiv for that matter, aid in the peer review process by disseminating pre-print cutting edge research findings thus allowing errors to be caught early on."

Posted by Maxine Clarke on July 20, 2007
Categories: Systems | Permalink | Comments (0)

From Nature's Correspondence section this week (Nature, 447, 1052; 28 June 2007)
Sir -- I was excited when Nature launched its trial for open peer review last year, but disappointed by the outcome . I have also been tracking the progress of another open review journal, Biology Direct (see article in Nature's peer-review debate ). Even after a high-profile launch with guaranteed indexing by PubMed, this journal has published only 52 articles and received only two comments over 16 months.
In contrast to Biology Direct, another journal that offers open commenting, PLoS One , has published 1,189 articles in its first six months. But has PLoS One achieved its stated goal of post-publication open comments? I find that even the 'most annotated' category of articles usually receives just a few comments. The journal has recently replaced its 'most annotated' with a 'recently annotated' category. A check of all 'recently annotated' articles demonstrates that their commenting rates are low (zero or just a few), even for articles that are likely to have broad appeal and/or are in 'hot' research areas.
Why is there a general lack of interest among the scientific community in open commenting on submitted or published papers? I believe there are two main reasons. First, participation does not earn any tangible credit or benefit for the reviewers and commentators. Second, publicly critical comments are a risk for those who make them.
Shi V. Liu
Scientific Ethics, Apex, North Carolina

[Correction added by Maxine Clarke: Chris Surridge of PLOS One points out in a comment to the post that the journal published just over 550 articles in its first six months, not 1,189. Thank you, Chris.]

Posted by Maxine Clarke on June 29, 2007
Categories: Systems | Permalink | Comments (2)

Among the excellent content on Nature Reports Stem Cells is an especially exciting regular feature, "Inside the Paper", which zeros in on recent scientific papers in the field published in Nature and provides detailed expert comment and author responses drawn from the peer-review process. The editors intend this feature to let readers see the paper’s context, strengths and caveats, as well as make the peer- review process a little more transparent.

Two Inside the Paper features have been published this month. The first, Cloning from Chromosomes, discusses the Article by D. Egli et al., Developmental reprogramming after chromosome transfer into mitotic mouse zygotes, in Nature 447, 679–685 (2007). Read a panel of experts' comments on this interesting article, responses from the authors, and comment yourself on The Niche (the Nature Reports Stem Cells blog).
The second Inside the Paper (21 June), Rewriting in blood, is about a paper by I. M. Samokhvalov et al(Nature 446, 1056–1061; 2007), reporting a surprising origin for blood stem cells. As before, you can read some of the peer reviewer's comments, responses from the authors and invite you to add your contribution on The Niche.

Posted by Maxine Clarke on June 28, 2007
Categories: Quality and value, Systems | Permalink | Comments (0)

Nature Precedings is now out of "beta testing" and is launched. This new community service is described at Nautilus, the NPG blog for present and future authors. Submissions are screened by our professional curation team for relevance and quality, but are not subjected to peer review. High-quality contributions from biology, medicine (except clinical trials), chemistry and the Earth sciences are welcomed.

More details about Nature Precedings can be found here. Because they have not been peer-reviewed, many of the findings you read at Nature Precedings may be preliminary or speculative, and remain to be confirmed. Please bear this in mind when deciding how seriously to take them.

Submissions are not accepted from fields in the physical sciences that are are already well served by preprint servers such as arXiv.org. Content that considered to be non-scientific or pseudoscientific is rejected. We accept only genuine contributions from qualified scientists. This will usually require submitters to have a recognized academic affiliation. Incomplete submissions will also be rejected. This is a free service, so please help us to help you by completing all relevant sections of the submission form.

The reactions of scientists and publications to the launch of Nature Precedings can be found at this Connotea page, which is regularly updated. Here are one or two such articles:
Nature Precedings pre-print server for biomedical research
Nature Precedings: A nicer version of ArXiv[e] for biomedical research
Chemistry Central: A new preprint server from Nature.

Posted by Maxine Clarke on June 18, 2007
Categories: Open peer review, Systems, Technical solutions | Permalink | Comments (0)

Steven Novella of the blog NeuroLogica writes an essay on peer review. " Peer-review is a critical part of the functioning of the scientific community, of quality control, and the self corrective nature of science", he writes. "But it is no panacea. It is helpful to understand what it is, and what it isn’t, its uses and abuses."
The essay is a short but informative account of the standard peer-review processes of many scientific journals, including a short paragraph about post-publication peer-review. At the time of writing this post, there are two comments on the subject of peer-review and fraud.
Dr Novella's "bottom line: peer-review is a necessary component of quality control in science, but is no guarantee of quality, and you have to know the details of the journal that is providing the peer-review."

Posted by Maxine Clarke on June 14, 2007
Categories: Systems | Permalink | Comments (0)

See Nature Reports Stem Cells for a fresh perspective on a paper reporting a technique for cloning from zygotes. In this fortnight’s Inside the Paper, a new form of scientific reporting pioneered by Nature Reports Stem Cells, you can read a moderated discussion between the authors and the paper's peer-reviewers. Readers can learn what the foremost experts in the field had to say about the submitted paper’s strengths and shortcomings. See what the authors saw, and read their responses as they revised their paper for eventual publication in Nature. You can also add your own comments on The Niche, the Nature Reports Stem Cells blog.
Featured paper: D. Egli et al. Developmental reprogramming after chromosome transfer into mitotic mouse zygotes. Nature 447, 679–685 (2007).

Posted by Maxine Clarke on June 13, 2007
Categories: Systems, Technical solutions | Permalink | Comments (0)

Picking up on the Nature Structural and Molecular Biology Editorial "New data at conferences, please", discussed here last week, Mario Pineda-Krch writes on his blog (Mario's Entangled Bank) about how the idea of the conference as a type of open peer-review process reminds him of the 1997 essay The academic Cathedral and the Bazaar by Eric S. Raymond.

In the Cathedral model the research process occurs behind closed doors among a exclusive group of researchers where progress is reported in peer-reviewed publications. In the Bazaar model the research is conducted in full view of the public......The basic tenet of the Bazaar model has in the field of software development termed Linus' law, i.e. "given enough eyeballs, all bugs are shallow". In an academic research environment the pros of the Bazaar model hinge on the fact that the more widely available the your research methodology and results is for public testing, scrutiny, and experimentation, the more rapidly all forms of errors and omissions will be discovered. In contrast, in the Cathedral model an inordinate amount of time and energy must be spent hunting for errors due to the limited number of eyeballs.

Mario, a postdoctoral researcher who works on ecological population dynamics at the University of California, Davis, believes that "academia would be a better place if more people would embrace this type of openness. Maybe the conferences setting could be the natural stage where the Bazaar movement could start."

Posted by Maxine Clarke on June 12, 2007
Categories: Systems | Permalink | Comments (0)

Inherently Responsive is the blog of the journal Heredity, published by NPG. As explained at the post Your chance to respond, the journal is seeking to improve its peer-review process using the opportunities provided by the web. One proposed model is to publish papers accompanied by referees’ comments, another is to publish correspondence columns in the journal after publication of the paper, to air legitimate differences of scientific opinion.
Inherently Responsive combines aspects of these two approaches. It publishes rapid feedback on papers that have appeared in Heredity: technical comments, relevant work published elsewhere and so on. Here is one example, about inheritance of litter size in Arctic foxes. It also plans to publish comments provided for public consumption by the peer-reviewers, so readers can see why the paper is controversial or why publication was recommended.
Inherently Responsive also features discussion on the editorial direction of the journal. In large part the content of Heredity is determined by what is submitted, but some of it is are commissioned. The blog seeks to obtain journal readers' views on whether important areas are being negelected.

Posted by Maxine Clarke on June 12, 2007
Categories: Systems | Permalink | Comments (0)

NIH Establishes Working Groups to Examine Peer Review

The Director of the US National Institutes of Health (NIH), Dr Elias A. Zerhouni, annnounced on Friday last week (8 June) the formation of two working groups to examine the NIH peer review process, with the goal of maximizing its effectiveness. The two groups will seek input from the scientific community, including investigators, scientific societies, academic institutions and health organizations, as well as from within NIH. The groups will study the context, criteria and culture of peer review to make sure the most talented individuals and reviewers are engaged in the process. More details, and the composition of the external and internal working groups, are available at the link at the top of this post.

Posted by Maxine Clarke on June 11, 2007
Categories: Systems | Permalink | Comments (0)

Reviewing appeals : Article : Nature Immunology

This month's Nature Immunology editorial (8, 541; 2007 -- link above); which is freely available, describes the journal's peer-review process and how it deals with appeals against decisions not to publish a submitted paper. This process is similar in all the Nature journals that publish original scientific research. From the editorial:

We believe that constructive critiques made during the review process improve manuscript quality, whether the manuscript is ultimately published in Nature Immunology or in another journal. Clarification of the review and appeal processes, as provided here, should assist authors in their preparation of manuscripts. We hope that by offering this advice to authors we can lessen frustration should a negative decision be rendered.

The full text of the editorial can be accessed via the link at the top of this post. Comments are welcome.

Posted by Maxine Clarke on May 25, 2007
Categories: Systems | Permalink | Comments (1)

Notes from the biomass: Warnock’s dilemma is a blog post on the discussion about lack of comments on scientific papers that are opened up for reader feedback online. Mr Warnock has a Wikipedia entry (which can be found via the Notes from the Biomass link) explaining why he thinks lack of comments does not (necessarily) indicate lack of interest.

Posted by Maxine Clarke on May 17, 2007
Categories: Systems | Permalink | Comments (0)