Peer-to-peer

This post is by Myles Allen of the University of Oxford:
As a vocal supporter of the traditional system of scientists communicating through peer-reviewed channels ( Nat. Geosci. 1, 209; 2008 and associated debate at Peer-to-Peer), I was hesitant about writing a critical Commentary on the Feature 'A safe operating space for humanity' by Johan Rockström et al in the 24 September issue of Nature (Nature 461, 472-475; 2009) in a non-peer-reviewed forum. The Nature and Nature Reports Climate Change editors had clearly thought through this argument: the Feature was not itself peer-reviewed, so no golden rules would be broken in publishing a series of commentaries alongside it in Nature Reports Climate Change.
The problem is that packing the point into a few hundred words, and slipping into the usual bloggers' trap of feeling you have to shout loudly on the internet or no-one will listen, means that the end result reads very black-and-white. Corresponding with Johan Rockström over the past week, it emerges we agree on far more than the tone of my Commentary, 'Tangible targets are critical' (Nat. Rep. Climate Change doi:10.1038/climate.2009.95), probably implies. I understand that Rockström et al. had not originally intended to make the link between a six-degree climate sensitivity, the two degrees target and 350 p.p.m. a focal point (as I read it) of their Feature. Likewise, they observe, reasonably enough, that limiting cumulative carbon dioxide emissions to one trillion tonnes of carbon is just another way of framing the climate boundary, with (as I acknowledge) remarkably similar implications to 350 p.p.m.
If this had been done the old-fashioned way, Rockström et al. would almost certainly have had to qualify their reliance on a six-degree climate sensitivity in the course of the usual to-and-fro with referees. And I in turn would have toned down a lot of my objections. The end result would undoubtedly have been blander on both sides, but would that really have been much of a loss? Following what was, for me, an experiment, I still feel it is very much an open question whether scientific communication in general benefits from direct publication rather than allowing rough edges to be smoothed off through traditional peer-review.
Myles Allen is at the University of Oxford, Parks Road, Oxford, OX1 3PU, UK.
-----------------------------
Context:
Johan Rockström and his co-authors argue in a Nature Feature that to avoid catastrophic environmental change, humanity must stay within defined 'planetary boundaries' for a range of essential Earth-system processes. If one boundary is transgressed, then safe levels for other processes could also be under serious risk, they caution. Seven expert commentaries respond to this proposal in Nature Reports Climate Change, one of which is by Myles Allen. All these articles can be accessed from this index page.
There is an associated Nature podcast in which Johan Rockström is interviewed and editor Ehsan Masood provides further analysis.
Further reading:
A longer paper upon which the Nature Feature is based, 'Tipping towards the unknown', is available at the Stockholm Resilience Centre.
Nature news special: The road to Copenhagen.

Posted by Maxine Clarke on September 23, 2009
Categories: Quality and value | Permalink | Comments (3)

Manuscript peer reviewing is at the heart of the scientific system, but it seems that these duties are often not properly (if at all) recognized by universities, funding agencies or even the rest of the scientific community. This is the main message of the September Editorial in Nature Structural & Molecular Biology, 'The Unsung Reviewer' (16, 899; 2009) The Editorial notes:
"Scientists wear many hats these days. They prepare and teach classes and sit on various committees. Then there are the multiple activities directly related to their research, including grant writing, mentoring students and postdocs, attending conferences, writing papers and reviewing manuscripts. All these duties can take a considerable amount of time and effort and most are recognized as worthy contributions by funding agencies, universities and research institutions when evaluating a scientist's performance. On the other hand, peer reviewing papers seems to be the Rodney Dangerfield ("I get no respect!") of a scientist's duties."
A new report by the Science for Policy project of the US Bipartisan Policy Center notes that peer-review is essential for the scientific system. Journals such as NSMB (and the other Nature journals) would not agree from their own perspective with one of the report's conclusions, that "peer-review is no longer assumed to be a professional obligation". As the Editorial points out, our journals have very broad reviewer pools (running into many tens of thousands for the heavily oversubscribed weekly title Nature, for example), and the editors know and very much appreciate the work that these reviewers put into improving submitted manuscripts.
The SPP report also identifies a need to increase the number of scientists who participate in peer review for federal agencies, making suggestions such as listing such service on grant applications or even making it a requirement for funding. In addition, it urges journals to run a quality system, for example by providing the peer-reviewers with feedback (which the Nature journals do, and which is appreciated by our reviewers - for one reason, because a reviewer can see his or her report in the context of those of the other reviwers of the manuscript). The NSMB Editorial concludes:
"Peer review is often compared to jury duty, a chore that one has to do once in a while as a service to the community. But reviewing manuscripts can be enjoyable and useful at any career stage. True, it takes time, but as one reviewer aptly put it: "I still learn do's and don'ts from reviewing papers, as well as just getting useful information, so it is time spent pretty well." It is also true that one does not get much open recognition from it, and there are limitations on what the journals can do without compromising the reviewers' anonymity. We have in the past publicly acknowledged our reviewers at the end of the year, and will do so again this year. The SPP report stresses that it is important to stress that "peer reviewing manuscripts should be an expected and appreciated aspect of a scientist's career." Of course, we do have a vested interest in this, but we firmly believe that it's important to cultivate a vibrant 'reviewer culture'."

Nature journals' peer-review policies and guidelines.
More on peer review from the Nature journals.

Posted by Maxine Clarke on September 14, 2009
Categories: Quality and value | Permalink | Comments (0)

A guest post from Willy Aspinall
Department of Earth Sciences, Bristol University, Bristol BS8 1RJ UK.

The Nature Editorial (‘Experts still needed’, Nature 457, 7-8 (2009); free to access online) and Harnad’s related Correspondence item on research performance metrics in the 2008 Research Assessment Exercise (Nature 457, 785; 2009) prompt me to suggest that an additional, complementary metric is needed which would measure the accomplishments of research scientists who act as peer-reviewers for journals.
Good reviewing is very time-consuming and, in some ways, just as challenging as authoring an original research paper; time spent doing this well is time removed from one’s own research work. Indeed, the thoughts and comments of a good referee can often represent a fundamental contribution to the science as well as the quality of a published paper, and this input should be recognized, and measured (the American Geophysical Union regularly celebrates ‘excellence in reviewing’ with citations by its journal editors). It is probably fair to say also that tangible good performance in refereeing usually begets ever more requests to review even more manuscripts, with further incursions on the diligent and proficient scientist’s time.
Perhaps a metric for this essential scientific activity of peer-reviewing might be constructed by summing the number of papers refereed by the individual scientist per year, each review being multiplied by the Impact Factor of the journal concerned. As refereeing is usually a solo activity, a metric for this skill, and for the related professional commitment, would be less prey to the shortcomings of performance measurement associated with metrics that attempt to gauge multi-author citations, for instance. Combining a 'refereeing metric' with other citation-related metrics to obtain a more comprehensive performance score for an individual scientist should not be an insuperable problem – and this measure can be pooled, as indicated in the Nature Editorial, with expert evaluation.
Willy Aspinall

Posted by Maxine Clarke on April 07, 2009
Categories: Quality and value, Technical solutions | Permalink | Comments (5)

In 2008, the journal Nature Neuroscience joined a newly created community consortium aimed at making peer review more efficient by allowing reviews to be transferred between consortium journals. In its current (April) issue, the editors look back at their experience with the Neuroscience Peer Review Consortium over the past year (Nature Neuroscience 12, 363; 2009).
Journals in the Neuroscience Peer Review Consortium (NPRC) offer authors whose papers are no longer under consideration at a journal an opportunity to transfer reviews of their manuscipts when submitting their paper to another consortium journal. After a year, Nature Neuroscience's experience is similar to that of other journals in the consortium, with only a handful of papers being transferred from Nature Neuroscience to another consortium journal.
Similar to the Nature journals' transfer system, the NPRC system is voluntary for authors and referees. Editors at one journal only know that a paper was reviewed elsewhere if the author chooses to inform them. At Nature Neuroscience, the editors ask referees for permission to release their identities whenever authors ask for their papers to be transferred to another consortium journal. If a reviewer declines to participate, the reviews (comments to authors only) are transferred anonymously.
All the transfers from Nature Neuroscience to date have been to the Journal of Neuroscience, and represent less than 1% of manuscripts that are eventually rejected after review. However, for the papers that were eventually published in the Journal of Neuroscience, the authors reported that the paper had been expedited. Even in the cases where new referees were solicited, authors felt that transferring the reviews from Nature Neuroscience had saved them time and effort.
No papers have been transferred to Nature Neuroscience from other consortium journals.
The Nature Neuroscience editors ask why so few authors are using the NPRC option. They conclude: "Authors may simply not be aware of NPRC or may not know what journals participate in it. Transfer rates may pick up as more authors learn of the consortium. At Nature Neuroscience, we have noticed an increase in the number of referees that state in comments to the editors whether they wish for their identities to be released to other consortium journals or not, suggesting a growing awareness of the NPRC.
It could also be that there are not that many papers that lend themselves well to this process. Many of our authors who have had papers rejected may prefer to take their chances with new referees at another journal, rather than making substantial revisions in response to the concerns raised by our referees. Certainly, our authors appear to be more conservative when deciding to transfer their reviews, preferentially choosing to utilize the NPRC transfer option when the reviewers reject the paper on conceptual grounds and not for technical reasons.
Another factor that influences the success of the transfer is whether the referees allow the release of their identities to receiving consortium journals. Previous reviews are clearly less useful to the receiving journal if the editors do not know who the reviewers were."
Nature Neuroscience concludes that it is premature to gauge whether the system truly could save referees, authors and editors substantial time and effort. The editors encourage authors, referees and readers to share their views, either by email or by commenting here.

Posted by Maxine Clarke on March 31, 2009
Categories: Quality and value, Systems, Technical solutions | Permalink | Comments (0)

Brian Derby writes a post with the title Refereeing Chores at his Nature Network blog. Brian has refereed hundreds of papers in his scientific career, so he's as experienced as it gets at the process. Even so, he writes about how it took half a day to reach his decision - to recommend rejection of the submitted paper - and had yet to write his report in suitably critical yet constructive tones for the author and journal.
The process required a first read to form an initial impression, a second read, to use Brian's words, "required that I check up on some of the references and make sure that the article is consistent, which it was but I did not agree on the applicability of the model proposed. More reading of the bibliography. I discover at least one self-reference that appears to be there for no reason other than to (perhaps) improve citation score (I am now grumpy). I discover another reference that is on a completely different subject that must be in by mistake.... A discreet phone call to a colleague allows me to discuss the mechanism and we agree that it is flawed. Now follows the difficult bit, I have to write a carefully worded letter justifying my concerns to the editor."
Or to put it another way, here is how the Nature journals recommend a review is written:
"The primary purpose of the review is to provide the editors with the information needed to reach a decision. The review should also instruct the authors on how they can strengthen their paper to the point where it may be acceptable. As far as possible, a negative review should explain to the authors the weaknesses of their manuscript, so that rejected authors can understand the basis for the decision and see in broad terms what needs to be done to improve the manuscript for publication elsewhere. This is secondary to the other functions, however, and referees should not feel obliged to provide detailed, constructive advice to authors of papers that do not meet the criteria for the journal (as outlined in the letter from the editor when asking for the review). If the reviewer believes that a manuscript would not be suitable for publication, his/her report to the author should be as brief as is consistent with enabling the author to understand the reason for the decision.
Confidential comments to the editor are welcome, but it is helpful if the main points are stated in the comments for transmission to the authors. The ideal review should answer the following questions." (More than a dozen follow, and this list is followed by a second list of "further questions to consider".)

Posted by Maxine Clarke on March 10, 2009
Categories: Quality and value | Permalink | Comments (1)

"Scientific publishing depends on expert peer reviewers. Instead of perpetually arguing about the reliability and fairness of peer review, authors, editors and referees should seek to optimize this time-tested system." So opens the January editorial of Nature Neuroscience (12, 1; 2009).
The editorial discusses the media reporting of a ferocious argument about the merits of a paper published in Cell , and a subsequent blog debate hosted by The Scientist. The controversy between scientists in this discipline concerning this paper "has again ignited a debate on the flaws of editor-managed anonymous peer review", write the Nature Neuroscience editors. "We maintain, however, that despite occasional unfortunate lapses, anonymous peer review remains the best quality-control process that we have." The editorial goes on to discuss how journals can best optimize the process.
In Nature News this week (457, 245; 15 January 2009) another technical dispute is discussed, this time concerning a widely circulating preprint attacking much of the published research in social neuroscience involving functional magnetic resonance imaging (fMRI). This affair is exacerbated by the fact that the circulating preprint is not due to be published, with responses from the authors of some of the criticized studies, for another nine months, according to the Nature News story. The criticisms have already been covered in publications such as Newsweek, as well as the inevitable blogosphere outpourings - and at least some of the criticized authors say the first they heard of the preprint was when they were contacted by journalists.
How scientific reports should be peer-reviewed and, after publication, scrutinized are matters that are decided upon by the journals, their editors, and their publishers and/or societies - in the case of articles submitted to and published in the Nature journals, the peer-review process is described here, and the post-publication corrections process here. Good journals have processes for investigating technical criticisms and complaints about the papers they publish. Nature's, for example, is here. Often, a resolution is not clear-cut at the outset, when the complaint is first received by the journal, however clear it may be in the mind of the complainer. A proper outcome depends on independent peer-reviewers, as well as editors, examining the complaint together with a measured response from the study's authors. Playing out such investigations in the kangaroo courts of the popular press, or in unfettered comments on the Internet between people who have been described as "recreationally outraged", not only obscures logical, technically informed investigation, but unnecessarily exacerbates emotions and arguments so that, in the end, all that is remembered is the heat - not any light.

Posted by Maxine Clarke on January 16, 2009
Categories: Ethics, Quality and value, Systems | Permalink | Comments (2)

The Journal of Cell Biology has launched an application that is intended to transform the way researchers can store, share and access the growing number of microscopy images. (See E. Hill, J. Cell Biol. 183, 969–970; 2008). The service is described in a Nature online news story on 19 December 2008.
Cell and molecular biology laboratories routinely produce image files in a wide variety of formats, and researchers often keep images on their own hard drives. The new JCB DataViewer, supported by the University of Rockefeller Press, publisher of the journal, is a database into which all authors of papers published by the Journal of Cell Biology have to submit their raw images — which can amount to a terabyte or more of data. The software then allows others to retrieve and manipulate the data through a web browser. As well as the ability of the system to store these huge datasets, the hope is that the JCB DataViewer will help peer reviewers and others in the community judge the quality of the work.
From the Nature news story: "It's transparency and integrity," says Jason Swedlow, a microscopy specialist at the University of Dundee, UK, who led much of the technical development. "It is going to allow you to see that data and judge it for yourself." The journal's editors hope that "eventually a version of this database could become a central repository for image data from the worldwide research community — something like a microscopy version of Genbank, a database in which vast amounts of genetic-sequence data are shared and stored. In future, research funding agencies and journals might require that authors submit their microscopy data to such a repository, just as most require submissions of DNA sequences to Genbank or other similar banks. "There have been murmurings about wanting data to be kept and maintained somewhere but no one has mandated it yet," Hill says."

Posted by Maxine Clarke on December 22, 2008
Categories: Quality and value | Permalink | Comments (1)

A news story in Nature Medicine (14, 1133; 2008) discusses an investigation into the publication status of the clinical-trials literature (PLoS Med., doi: 10.1371/journal.pmed.0050191; 2008), which concludes that positive results of clinical trials for drugs or devices have a higher chance of getting published than negative trials. The study further concludes that when the trial sponsors publish the results from 'pivotal' trials only 76% of the time.
Studies reporting a statistically significant difference were more than three times as likely to be published. This 'positive publication bias' is a serious problem, because it can make a drug or device appear in the literature to be more effective than it is. Ida Sim, a co-author of the study, told Nature Medicine: "We have this idea of practicing evidence-based medicine, which is predicated on having a full and complete evidence base. But when the evidence base is skewed, we can't really do this."
According to the Nature Medicine article, a paper in Science (319, 1340–1342; 2008) indicates that "the FDA Amendments Act of 2007 has improved transparency, because the law mandates that sponsors or primary investigators of clinical trials for approved drugs post a summary of their results in a national open-access database. The lead author of the report, Deborah Zarin, oversees the ClinicalTrials.gov registry at the National Library of Medicine of the US National Institutes of Health and is in charge of ensuring the results are posted in compliance with what the new law. According to Zarin, "for the trials that are covered by this law, the results database should have a big impact on disseminating medical knowledge, because the results have to be publicly available."
But not every type of clinical trial is covered by the legislation, nor does it directly affect medical journals. Although Sim applauds the FDA Amendments Act of 2007, she adds that it "doesn't address the issue of not publishing trials in medical journals. They remain one of the most influential and biased sources of information." "

Posted by Maxine Clarke on November 27, 2008
Categories: Quality and value | Permalink | Comments (0)

The Editorial 'Working double-blind' (Nature 451, 605–606; 2008), also republished on this blog and stimulating more than 70 comments, referred to a study (1) that found more female first-author papers were published using a double-blind, rather than a single-blind, peer-review system. The data reported in ref. 1 have now been re-examined (2). The conclusion of ref. 1, that Behavioral Ecology published more papers with female first authors after switching to a double-blind peer-review system, is not in dispute. However, ref. 2 reports that other similar ecology journals that have single-blind peer-review systems also increased in female first-author papers over the same time period. After re-examining the analyses, Nature has concluded that ref. 1 can no longer be said to offer compelling evidence of a role for gender bias in single-blind peer review. In addition, upon closer examination of the papers listed in PubMed on gender bias and peer review, we cannot find other strong studies that support this claim. Thus, we no longer stand by the statement in the fourth paragraph of the Editorial, that double-blind peer review reduces bias against authors with female first names.
References
1. Budden, A. E. et al . Trends Ecol. Evol. 23, 4–6 (2008).
2. Webb, T. J. , O'Hara, B. & Freckleton, R. P. Trends Ecol. Evol. doi:10.1016/j.tree.2008.03.003 (2008).

Posted by Maxine Clarke on June 05, 2008
Categories: Ethics, Quality and value, Systems | Permalink | Comments (2)

Ignazio R. Marino* writes in Correspondence in the current issue of Nature (453, 449; 22 May 2008):
'Italy must invest more in science and technology' according to I. Bertini, S. Garattini and R. Rappuoli in Correspondence (Nature 452, 685; 2008). They lament the Italian lack of financial resources and political attention for research, technology and education. As a researcher, clinician and academician, I share their concerns. However, as former chair of the health committee of the Italian Senate, I take exception to their implication that none of the major political parties recognizes science, technology and education as crucial for the future of the country's economy.
The 2007 and 2008 national budget laws, drawn up when the centre-left coalition was in power, allocated 96 million (US$149 million) to projects submitted by researchers under 40 years old. These are judged by an international committee comprising ten scientists under 40 — five from foreign institutions — selected according to impact factor and citation index scores. This alone is a revolutionary approach for the unregulated Italian system of research funding allocation.
In spite of such advances, Italy is still far behind in research investment, and this needs to change. But the crucial switch is not simply to increase funding. The way the new government should proceed is to reform the allocation criteria for funding and to start applying across the board the selection and evaluation rules of peer review. Such a system would acknowledge meritocracy and free researchers from the virtual slavery under which they have been kept by old academicians.
By applying international rules of peer review and evaluating grant applications only on the basis of merit, looking at curricula and objectives, comparing lists of publications and evaluating results, we will provide opportunities for Italy's scientists, thereby promoting the country's intellectual, cultural and economic growth.
*Department of Surgery, Jefferson Medical College, 19107 Philadelphia, Pennsylvania, USA, and Senate of the Republic of Italy, Piazza Madama snc, 00186 Rome, Italy.

Posted by Maxine Clarke on May 27, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)

From an Editorial (free to access online) in last week's Nature (453, 258; 15 May 2008):

At first glance it seems to be a shining example of the scientific method in action. Two papers published by biochemist Homme Hellinga and his students at Duke University Medical Center in Durham, North Carolina, claimed a breakthrough in rational enzyme design. Last year, another chemist found that Hellinga's enzymes didn't actually work, which led to the retraction of the two papers this February (see page 275 of this issue and an earlier story on 13 February). Then, this March, a third group published research showing that rational enzyme design really is possible. All has ended happily, it seems, with the field marching forward in triumph.
But examined more closely, the episode reveals some less than happy aspects of science as it is actually practised. For example, the problems with Hellinga's enzymes were identified by John Richard at the State University of New York in Buffalo, who hoped to use the proteins in his own work. In effect, Richard and his two co-workers wasted seven months and tens of thousands of dollars failing to reproduce the results from Hellinga's lab. Richard's subsequent efforts to correct the scientific record thus came at considerable cost, with no discernable benefit to his own career.
This is a perennial problem in science. Many researchers who come across non-reproducible work save themselves extra hassle and money by simply not pursuing it further. Meanwhile, those who refuse to let it go — like Richard — gain nothing.

Continue reading "Hellinga story exemplifies weaknesses of the scientific process" »

Posted by Maxine Clarke on May 21, 2008
Categories: Quality and value | Permalink | Comments (0)

This post is a continuation of the discussion about blogging and peer-review by selected reactions at RealClimate (a climate scientists' community blog) to Nature Geoscience's two commentaries on blogging.

Philip Machanick writes that the "problem with blogs is that there is no way for an outsider to know which are reasonably careful creations of informed scientists, which are opinions of the scientifically illiterate, and which are astroturf creations designed to confuse critics of science that is in conflict with an industry. Given that terrain, I would rather have something like RealClimate than not: it helps to balance things out. Errors tend to be corrected quickly here as a consequence of a large informed readership (even if it is sometimes annoying that you get drive-by ignoramuses who don’t benefit from getting their misconceptions answered). He goes on to suggest that a site such as RealClimate is of value in providing a forum for rebuttals of peer-reviewed science in order to develop a consensus on whether a "formally published rebuttal is worth the effort".

Ray Ladbury's view: "When it comes to peer reviewed papers, one has to presume the reader will have a minimum level of familiarity with the subject matter. One also presumes that the reader will have a day job, and so the question becomes whether the information in the paper is of sufficient interest to the average scientist in the community to say, “Hey, take a look at this. It looks mostly correct to me and has some interesting information/insights/methods…” This is not in any way the gold standard in the sciences. The gold standard comes when the community as a whole says, “Hey, cool, I can use this.” The paper is cited. The techniques are used. Science advances. Eventually, what was in the paper becomes part of the tacit knowledge assumed by reviewers.
The tacit knowledge one can presume for a blog like Realclimate is much lower. One presumes there is an interest in the subject–why else would the reader be perusing the blog. One presumes at least a passing acquaintance with the scientific method and maybe some familiarity with basic results like conservation of energy, etc. One could perhaps assume that the average reader has taken the time to acquaint him- or herself with material to which one is vectored via the “Start Here” button–although this is far from Universal.
For the average newspaper reader of a science story, the tacit knowledge is nearly nonexistent–or worse, wrong. And then we have the blogosphere, where information density is at best, rarified and often toxic.......in an information economy, it seems that all too many readers and journalists are content to remain paupers."

Posted by Maxine Clarke on April 29, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)

In a comment to an earlier Peer to Peer post, 'Regular Scientist' takes issue with Nature Cell Biology's definition of a good report, writing (slightly edited for length): the journal sometimes asks the authors to undertake many experiments even before sending a paper to review. Not strangely, papers in Nat Cell Biology, even small reports, contain many Supplementary Figures (up to 20, I´ve seen). I miss the old days where publications were for sending a very interesting result so you could extend it and discuss its importance for your field. Nat Cell Biology's system is undermining research efforts in the authors´ labs, which have to dedicate an enormous effort to provide many additional experiments that don´t even add much to the concept. This is not what science, and the peer-review process, should be about. I would recommend the editors of Nat Cell Biol a calm reading of the original paper on the structure of nucleic acids, and note the way it was written.

Bernd Pulverer, Chief Editor of Nature Cell Biology, has kindly agreed to my request to write a guest post to respond, and to elaborate on the journal's practices and policies:

As ‘Regular scientist’ points out, we do strive to publish rather well-developed studies in Nature Cell Biology. It is interesting to read the concern about the supplementary information, since we usually hear the opposite complaint: namely that the tight format restrictions applied preclude presentation of complete datasets. In considering submissions, we expect fewer details of molecular mechanistic analysis if the paper reports a highly novel observation. And because the study of cell biology tends to throw up surprising new findings that deserve a wide audience but that cannot be developed in a reasonable timeframe, we decided some years ago to publish a short ‘Brief Communication’ format, which I hope is appreciated by Regular Scientist.
In the Editorial under discussion, we wrote:

Nevertheless, the most important part of the report is assessment of the data: are key experiments or crucial controls missing? Are the data significant and definitive? Are all claims made supported by the data? A black and white model should not be a requirement — in fact it often underestimates biological complexity. Formulate a clear set of recommendations for additions or improvements. Experimental suggestions are important even if an outright rejection is recommended, as it makes for a transparent and constructive report that will allow the author to improve the dataset and select an appropriate target journal. If the overall assessment is positive, do not feel obliged to ask for non-essential experiments. New experiments should add key information, buttress claims or improve existing data, and they must be achievable. Indeed, the perception that referees invariably ask for more data encourages some authors to hold back data in the hope that these will be requested and can then be provided by return post; clearly, this is a futile exercise. Referees and editors alike should take great care that all key issues are raised in the first assessment — draw a line and adhere to it. Subsequent evaluations may raise issues on new data, but raising yet more ways to develop a study is not fair.

We, the Nature Cell Biology editors, agree that one can always ask for more data, and that sometimes data require an unrealistic amount of effort for the information added. However, we entirely disagree that our editorial and review process does not result in the publication of much-improved datasets. The tendency of referees to feel an obligation to ask for some new data, and conversely for authors to hold back data to give referees something obvious to ask for, is an issue that we tried to address in the Editorial.
We certainly do not ask for more data because we, the editors, feel under any compulsion to do so: the simple fact is that a good number of manuscripts submitted to the journal are rather premature, and we cannot justify wasting our referee’s time when we perceive obvious holes in a dataset that the referees would invevitably want to be addressed. We believe that this step is in an author’s interest, since we will not undertake multiple rounds of peer-review without good reason. If requested data are deemed to be unobtainable for technical reasons, or indeed unnecessary, we are always willing to hear an author’s arguments.
Regarding the 1953 Nature paper that Regular Scientist recommends: indeed, the paper is a good read. However, science has become much more complex since then, and the discoveries often more detail-oriented, requiring a more in-depth analysis. At the same time, the tools available have developed tremendously in 55 years. Even in 1953, readers will note that Watson and Crick's conclusions depended on the two following papers in the issue, providing experimental details. The challenge is to set ‘the bar’ correctly to what is achievable and what should be achieved for publication in a top-flight journal. This is what the job of an editor is all about.

Posted by Maxine Clarke on April 23, 2008
Categories: Quality and value | Permalink | Comments (1)

Continuing the discussion of the relative contributions of blogs and the peer-reviewed literature to scientific understanding, I'm highlighting another reaction to the two Nature Geoscience Commentaries presenting different perspectives.
In the comment thread at Real Climate blog, Simon Donner of the University of British Columbia writes:

"Blogs have created a forum for many people to informally discuss science. They are also a great forum for scientists to provide context to their own work and to the work of others in their field.
And some of the time, that may take the form of criticism. That’s ok. But we do have to be careful in blurring the line too much between peer-reviewed publications and blogs. The first problem is the obvious one. Blog posts are unfiltered, un-reviewed, and often written off the cuff, while journal articles are screened, reviewed, and (should be) meticulously researched. It is far easier to write a criticism of a paper on one’s blog than to write a response and submit it to the journal. The issue isn’t just that no reviewers check the work… the blog author is unlikely to do even close to the amount of research and analysis nor give the wording nearly the same level of consideration as is expected in a paper submitted to a high-quality, peer-reviewed journal.
The other problem is that blog posts are readily accessible to anyone at anytime, both in language, and in the unlicensed nature of the internet. To a huge swath of the public, blog posts are the face of science. Like it or not, bloggers with scientific credentials are like self-appointed ambassadors for science. If we are going to write about our science, we should do it with thought, and we should do it well. That is a standard that myself and many other science bloggers often struggle to meet."

Posted by Maxine Clarke on April 16, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)

The two Nature Geoscience Commentaries expressing opposing views on blogging's role in science research communication have been much discussed in the blogosphere (see Climate Feedback blog, for example.). One discussion took place at RealClimate blog, where Gavin Schmidt's post about his Nature Geoscience view on how science blogs and traditional peer-review intersect has attracted more than 100 online comments from climate scientists and others.
One commenter, "Tamino", writes: "Blogs don’t serve very well for communication among scientists. Peer review does more than just protect us from being inundated with substandard work; it protects authors from their own mistakes and improves the quality of what we write. Peer review itself is an immensely valuable avenue of communication; who among us hasn’t at some time included a phrase like “We thank an anonymous referee for comments and suggestions which dramatically improved the final manuscript”?
But as bad as blogs are for actual research, peer-reviewed journals are far worse for communicating with and educating the lay reader. Yet when it comes to climate science the lay public is hungry for knowledge, and many of them are eager, and well-prepared, for a level of sophistication and detail that can’t be found in lay journalism or even popular literature; An Inconvenient Truth isn’t enough. So blogs serve an incredibly useful purpose, enabling the interested and well-educated reader to share insights with researchers who are at the cutting edge of new knowledge.....................while blogs aren’t part of the machinery for legitimate scientific research, they’re an indispensible tool for communication and combating misinformation."

More to follow.

Posted by Maxine Clarke on April 10, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)

Nature Cell Biology continues its coverage of peer-review in its (free access) April Editorial 'Good review' (Nat. Cell Biol. 10, 371; 2008).
After stating the rationale for peer-review, the Editorial asks whether reviewing is essentially an altruistic activity, or whether there are significant rewards for doing it. "The biggest benefit of refereeing is probably to stay engaged with the scientific community in a sort of 'real-impact journal club'. We review less than a quarter of submitted manuscripts, with the aim of involving referees only for studies that seem worthy of publication. This saves time for authors, and referees know that the manuscripts they receive meet a minimum standard, so that refereeing should be an enriching experience. Nevertheless, the time commitment for refereeing is considerable and we are campaigning to ensure that this is taken into account in assessing research performance."
Although much has been said about alternatives and enhancements to 'single-blind' peer review, what actually constitutes a good referee report? The rest of the Editorial provides some answers, which are further explored at our author and peer-reviewers' website.
Nature Cell Biology's previous editorial (March 2008) is discussed here.

Posted by Maxine Clarke on April 04, 2008
Categories: Quality and value | Permalink | Comments (2)

Scientists know much more about their field than is ever published in peer-reviewed journals. Blogs can be a good medium with which to disseminate this tacit knowledge, according to Gavin Schmidt of the NASA Goddard Institute of Space Studies Columbia University, and co-founder of RealClimate.org, in a Commentary in the April issue of Nature Geoscience (1, 208; 2008). Dr Schmidt asks "why read a blog when you can go directly to the scientific literature? Unfortunately, access to new findings in the traditional way is harder than it should be. Many technical papers are behind pay-walls, which make it impractical and expensive for unaffiliated individuals to read them. More importantly, however, even when papers are freely available, they often do not provide the insight expected."
In another Commentary in the same issue of the journal (Nat. Geosci. 1, 209; 2008), Myles Allen of the University of Oxford agrees that "explaining science to journalists and the public on blogs is fast and efficient." But after his own bruising experience, he wonders: "is it all just too good to be true? Can science survive Web 2.0?"
After describing his own difficult experiences of having his own peer-reviewed work criticised and misinterpreted by people who would not subject their own conclusions to peer-review, Dr Allen concludes: "Everyone agrees we need to communicate science better to the general public. But more and faster should not be confused with better. I'm certainly not advocating closing science blogs or discouraging science websites. We just need to remember the basic courtesies that our doctoral supervisors took for granted: criticism of peer-reviewed results belongs in the peer-reviewed literature. Direct communication over the Internet, far from creating a level playing field, just ploughs it up and makes the game impossible. The problem is, without witty and cutting criticisms, what is the point of a blog? Sure, the peer-review system is creaking. Sure, science journalism sometimes trips up. But like Churchill's quip about democracy, it is the worst possible system for communicating scientific results, apart from all the alternatives that have been tried from time to time."
Yet Dr Schmidt remains optimistic: "Some may dismiss blogs as being a distraction from real scientific work, or of egging on the very controversies that we seek to diffuse. There is an element of truth to both of these claims. But the response should not be a return to the ivory tower. That simply leaves the field clear for those who prefer to confuse rather than enlighten. With the importance of science in policy decisions being more apparent than ever, our ability to do science and enhance its relevance in public life relies on the community's willingness to engage, inspire and inform. Blogs are one way to do that, and they can excel at providing the context that is so often missing in other media. Not every scientist needs to have one, but maybe every scientific field does."

Posted by Maxine Clarke on April 01, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)

Nature Neuroscience is joining a consortium of journals that enables reviews to be transferred from one journal to another, while allowing authors, referees and editors to control their degree of participation in the system flexibly. The reasons for the decision are explained in this month's (April) Editorial (Nat. Neurosci. 11, 375; 2008). Briefly, in January, a group of editors, supported by the Society for Neuroscience, implemented a system for transfer of submitted manuscripts between journals that allows voluntary participation by authors, referees and editors, known as the Neuroscience Peer Review Consortium. This consortium reduces the overall reviewing workload of the community by allowing authors to continue the initial review process when their paper moves from one consortium journal to another, once the paper has been rejected or withdrawn from the first journal. This arrangement is similar to the manuscript transfer system that has been available within the Nature family of journals , and all the other journals published by Nature Publishing Group, for almost a decade.
The neuroscience transfer system, described in detail in the Editorial and at the Neuroscience Peer Review Consortium website, is voluntary for authors and peer-reviewers. Journal editors have full discretion in deciding how to use transferred reviews. The receiving editor may choose to accept or reject a paper based on these reviews, without further consideration; to send the paper to some or all of the previous referees for evaluation of the authors' revisions; or to request a fresh set of reviews from new referees. Only comments to the authors are transferred to the receiving journal. Confidential comments to the editors are not passed along. Thus, to ensure transparency in the review process, both at Nature Neuroscience and at other journals after the paper has been transferred, journal editors encourage referees to include all their concerns about the paper in comments to the authors. According to the Nature Neuroscience Editorial, the small amount of extra time required to phrase comments diplomatically for the authors should be more than counterbalanced by the resulting improvement in the peer-review process.
Many members of the community have strong views on the issue of confidential comments, which can be found on the Action Potential blog, and which has previously been discussed at Peer to Peer. The success of the neuroscience journals' transfer system will be evaluated at the end of this year. Nature Neuroscience editors will also be evaluating the journal's participation on an ongoing basis, so they encourage authors, referees and readers to share their comments with the editors, either on Action Potential blog or privately by e-mail.

Posted by Maxine Clarke on March 31, 2008
Categories: Ethics, Quality and value, Systems | Permalink | Comments (0)

How many papers does the typical researcher review in a year? How long do they take? And why do they do it? For the answers, read the Editorial in this month's (March) issue of Nature Nanotechnology, "Who'd be a referee?" (3, 119; 2008). The Editorial reports some of the findings of the recent Publishing Research Consortium report on peer-review (previously discussed at Peer to Peer), whose survey revealed that the "average review takes about 8.6 hours (with a median of about 5 hours) and is completed within 3–4 weeks, although there are significant differences between the four broad subject areas covered by the survey: in the physical sciences and engineering, for instance, the average (mean) is 10.4 hours, compared with 6.3 for clinical researchers. Of course, on top of this, many researchers spend considerable time reviewing grant applications for funding agencies, which can be equally onerous and possibly even more important than reviewing papers."
For similar statistics, and a more general discussion of the peer-review process in publishing, see the Nature Nanotechnology Editorial, or see here for Nature Cell Biology's take.

Posted by Maxine Clarke on March 10, 2008
Categories: Quality and value, Systems | Permalink | Comments (0)

Much attention is being given to a Commentary in the current issue of Nature (Nature 451, 379-399; 2008), A tale of two citations, some of which I have attempted to encapsulate in this Nautilus post, for those interested. Although the issues immediately concern a possible increase in duplicate publication and plagiarism, as detected by software systems and database searches, peer reviewers are an integral part of the check/balance procedures that journals use. For this reason, I thought it well worth highlighting here the comment by Brian Derby at the Nature Network forum currently discussing these questions. Part of Dr Derby's response:

"As a referee I have identified duplicate or severely overlapping content while reviewing papers in the past (for reasonably high profile/impact factor journals). I do not search for duplication routinely but, as someone who is used to referee papers in particular niche areas, I received both papers in one instance and in another I had read an on-line pre-pub before receiving the duplicate. The authors will not be named as that would break referee confidentiality but they were from well known institutions in the developed world.
What was the common factor (apart from the paper!) – the authors were relatively junior new appointments. Younger academics seem to feel themselves under a lot of pressure to publish. In my department I believe that my younger colleagues are much more sensitive to impact factor than is possibly healthy when they consider where to publish an article."

Posted by Maxine Clarke on January 28, 2008
Categories: Ethics, Quality and value, Systems | Permalink | Comments (1)

The Publishing Research Consortium publishes a study this month (January 2008) in whch more than 3,000 senior authors, reviewers and editors were asked about the peer-review system. The conclusions are that researchers want to "improve, not change, the system of peer review for journal articles". According to the report, a summary of which is available (1.7 MB; PDF), more than 93 per cent of respondents believe that peer review is necessary, and more than 85 per cent say that it helps to improve scientific communications and increases the overall quality of published papers.
Although many respondents pointed out the operational difficulties in double-blind peer review, two-thirds of respondents felt that it is the most objectively fair system, compared with single-blind (the current prevalent system). Alternatives such as post-publication and open peer-review were not popular.
While of the majority of respondents saw peer review as an effective filter for research, some did not think it was effective at detecting plagiarism, fraud or misconduct. Interestingly, most reviewers among the respondents thought that paying peer-reviewers would be too expensive for publishers; most of them said that they perform reviewing as part of their support to their research community.
The full report is available here (1 GB; PDF). According to the Publishing Research Consortium, the main objective of the study was "to measure the attitudes and behaviour of the academic community with regard to peer review. This will inform debate concerning peer review, and underpin discussions, either in discussion lists or at future workshops/conferences."
This new report comes as the NIH (National Institutes of Health) finish analysing the thousands of responses to their assessment of grant peer review. Lawrence Tabak and colleagues are filtering the list into a set of key recommendations, which will be given to Elias Zerhouni, director of NIH, at the end of February.
Update, 29 Jan 2008. Nature Neuroscience discusses the NIH peer-review exercise in its February issue Editorial (Rethinking grant review Nature Neuroscience 11, 119; 2008).

Posted by Maxine Clarke on January 25, 2008
Categories: Quality and value, Systems | Permalink | Comments (0)

In an interesting exercise in how a journal can develop criteria for peer-review in a fast-moving, area of considerable intrinsic uncertaintly, the editors of Nature Reports Stem Cells posted an article last month that asked how one could declare human cells pluripotent, when the most robust tests are neither ethical nor feasible, calling on the US National Institutes of Health (NIH) to set standards. Now, in a post on The Niche blog, Defining pluripotency in human cells, the editors post some of the responses from researchers in the field. Here is one perspective, from Shinya Yamanaka of Kyoto University:

"This is an important, but difficult question. First of all, we don't know whether human ES cells are really ES cells or not. Because the lack of chimera experiments, we will not be able to answer this question. This means we lack a positive control. I have been telling my students that one of the worst experiments you can do is one without positive and negative controls."

The Niche also features an interview with Story Landis, head of the stem cell task force at NIH, about the world's response to the recent breakthrough result that human cells can be reprogrammed, including NIH's problem about how to develop criteria for funding in a research area that in itself does not support the US President's policy.

I will not be posting again on Peer to Peer until the New Year, so I wish you all a very happy Christmas and holiday season. If you feel the need for some interactive scientific discourse during this period of traditional peace for those involved in journal production, I can recommend Nature News (you can comment online on the articles) and Nature Network, where there are a range of groups, blogs and forums to suit any science-related interest, and to which you are welcome to contribute.

Posted by Maxine Clarke on December 19, 2007
Categories: Quality and value | Permalink | Comments (0)

In his Nature Network summary of a recent publishing meeting, Yiorgos Apidianakis describes his opinion that a peer-review "score" is a more desirable (and efficient) indicator of scientific excellence than the currently used impact factor of the journal that publishes the work. Charles G. Jennings's wrote, as part of Nature's peer-review debate, “It is common to bemoan the over-reliance on quantitative markers such as impact factors for assessing scientists’ abilities (and indeed there is much to bemoan), but until committee members have time to read every paper on every applicant’s CV, they will have to rely at least in part on proxy indicators.” Dr Apidianakis believes that an ideal indicator would be a score from a unified peer-reviewing system, or a central agency "that will thoroughly, rigorously and objectively evaluate any given work to be published, using again specialized scientists as reviewers. Having an evaluation score from such an agency, scientists can include this score to their publication record."

Dr Apidianakis thinks that such a system could work in practice, using the example of the NIH evaluation scoring system -- while admitting it would be very expensive. Irrespective of this practical obstacle (as well as others) I wonder how the quality of the agency's "score" can be standardized? The establishment of, in effect, one giant journal of research, with all the world's scientists signed up to it as peer-reviewers, is a stimulating concept. But, given the many divergences of views within fields, how could a centralized scoring system work? The IPCC has attempted a similar kind of approach for one discipline, climate change -- even though most scientists in the field broadly agree with the IPCC's assessment of research output, this consensus requires massive bureaucratic baggage, including many international meetings and vast reports justifying decisions. Yet there is a substantial minority of scientific dissentors, and many members of the public are sceptical of a unified approach resulting in "science by endorsement".

The current peer-review system works very well "in miniature", whether as operated by those journals able to call on the most thoughtful scientists in a field who do the most cutting edge research, or by literature reviews or research ranking services, usually written or operated by one or a few individuals. Scientific research itself has benefited from large author collaborations across the whole spectrum of disciplines, from astronomy and nuclear physics to genomics and cell signalling, facing the challenge of making sense of vast quantities of data. Are there the same intellectual and innovative advantages to be gained by a single-managed peer-review system for all of science?

Posted by Maxine Clarke on December 04, 2007
Categories: Quality and value, Systems | Permalink | Comments (0)

In a blaze of colour on the 11 November 'op-ed' (invited opinion) page of The New York Times, some scientists proclaimed that, based on analysis of brain-imaging data from just a handful of swing voters, they had divined what the rest of the undecided masses truly think about the upcoming US presidential elections. Apparently just asking them was simply not good enough.

So opens an Editorial in the current issue of Nature (450, 457; 2007), which goes on to describe how the authors of the New York Times piece used functional magnetic resonance imaging (fMRI) to scan the subjects' brains while looking at pictures of candidates. The Nature Editorial goes on:

"The op-ed work has not been published in a peer-reviewed journal, and the article is self-evidently too insubstantial in scientific detail to assess the strength of either the methods or the data. A group of cognitive neuroscientists was swift to object to its conclusions — which veer close to a modern-day phrenology — in a response to The New York Times.
The results described in the op-ed are apparently the claims of a commercial product posing as a scientific study. This is only partially transparent. Three of the authors list their affiliation with FKF Applied Research, a company based in Washington DC that is notorious for using similar brain-scan analysis to conclude which TV adverts aired during a major sporting event were most effective. In its own words, the company is a "business intelligence firm selling fMRI brain scan-based research to Fortune 500 companies".
Articles on The New York Times op-ed pages are opinionated by definition, and shouldn't normally require peer review. But here, the paper's editors have instead published the results of (to put it mildly) questionable scientific research, disseminating this information to millions of their readers who may not have the background to recognize for themselves the absurdity of some of the authors' conclusions.
Although it is a gross disservice to science and indeed to politics, it is a great deal for the company. Scientific publication would have required the authors to divulge their data and qualify their assumptions — and some journals might even have required that they declare their financial interests. Whatever the motives, seducing The New York Times' editors with the allure of Technicolor brains lighting up with Hillary Clinton angst yielded no more or less than a multimedia advertisement for the company's product to millions of readers."

Posted by Maxine Clarke on November 26, 2007
Categories: Ethics, Policy and Public, Quality and value | Permalink | Comments (0)

In an Editorial in today's Nature (450, 457-458; 2007), Replicator review, the journal describes its publication of "what we expect to be the final word on whether nuclear transfer can work in a primate — a paper by J. A. Byrne et al. showing not only that it is possible to clone primate embryos by somatic-cell nuclear transfer but also that precious embryonic stem cells can be derived from the embryos. If embryonic stem cells live up to their promise, the technology could be used to derive patient-tailored stem cells."
The Editorial goes on to describe how Nature took the unusual step of soliciting an independent verification of the paper during the process of peer review. "This is the first time that Nature has obtained second-party replication ahead of publication. It should not be seen as reflecting a mistrust of scientists in the cloning field or scepticism about this particular research group. Rather, our actions fulfilled a statement in an Editorial from 2006 that was conceived in the aftermath of the Hwang affair: "Keeping in mind the principle that extraordinary claims require extraordinary proof, Nature may in rare cases demand it."
Nature will continue to evaluate the need for such validation on a case-by-case basis. Meanwhile, researchers should consider maintaining tissue samples, and trying to establish and fulfil requirements that may go beyond the routine, as Byrne et al. did for their cloning paper. In that way, remarkable scientific developments can remain as free as possible from unwarranted speculation and controversy."

Also in this issue of the journal is a News and Views article on the new work, by Ian Wilmut and Jane Taylor (Nature 450, 485-486; 2007); a News story featuring an interview with one of the authors of the paper; and a Nature editors' one-paragraph summary of the work.

A blog discussion of peer-review challenges in stem-cell research and the peer-reviewers' reports for the paper by Byrne et al. can be found at Nature Reports Stem Cells.

Posted by Maxine Clarke on November 22, 2007
Categories: Quality and value | Permalink | Comments (0)

Bloggers for peer-reviewed research reporting, or BPR3, was conceived by scientists and others who write informally about research on the Internet as a simple way to denote that a blog post or website article is discussing peer-reviewed work. Their mission statement: "Bloggers for Peer-Reviewed Research Reporting strives to identify serious academic blog posts about peer-reviewed research by offering an icon and an aggregation site where others can look to find the best academic blogging on the Net."
As well as creating an icon for bloggers to denote when their posts concern peer-reviewed research, the organization will host a central web site where snippets from these posts will be displayed, with links back to the original posts. Readers will be able to choose topics of interest and view only those posts, if they wish.
Eventually, the intention is that bloggers will be able to enter a DOI (digital object identifier) or other unique identifier, and automatically generate code to post the icon, link to the post to the BPR3 site and its aggregation tools, and generate a properly formatted research citation which links to the original article.
Now, the first stage is complete -- the icons have been created (with a lot of helpful input from large numbers of science bloggers). They must be popular, as the site from which people can collect their code has been down for a couple of days due to excessive traffic, presumably. However, it is now up and viewable, so if you want to pick up an icon for your own blog, or just find out more details of the project, please visit this posting. From this post (29 October): "Anyone can use these icons to show when they're making a serious post about peer-reviewed research, rather than just linking to a news article or press release. Within a month, these blog posts will also be aggregated here, so everyone can go to one place to locate the most serious, thoughtful analysis and commentary on the web."
Inevitably, given the engagingly self-referential and gossipy nature of blogging, release of these icons attracted a lot of comment and discussion in the blogosphere. Dave Munger, the driving force behind BPR3, has collected links to all these articles into one post here.
Perhaps in some of this discussion, this question has been addressed already, but as I am afraid I do not have time to read all of the 26 (so far) articles, I raise it here. The icon seems to me an excellent way to indicate that the subject of a blog post is a peer-reviewed research article (so long as there is a mechanism to report abuse and remove the icon from posts that use it incorrectly). And the intentions of BPR3 in providing links to the original research article being discussed are admirable. However, these indicators in themselves do not seem to me necessarily to be an indicator of quality of the blog post itself. On a blog, anyone can write anything about anything, whether or not the topic under discussion is peer-reviewed. A lot of traffic to, or comments on, a blog post is not in itself an indicator of quality. (It could, indeed, be the opposite.)
After the blog posts are themselves linked to the original research article, it will be possible for them to be formally cited, making them eligible for aggregation into citation databases. Blog posts that enter the mainstream scientific debate in this sense would have a quality indicator associated with the post itself rather than the research they describe. Is this the intention?

Posted by Maxine Clarke on November 02, 2007
Categories: Perspectives, Quality and value | Permalink | Comments (4)

The Nature Precedings forum on Nature Network is featuring a stimulating and thoughtful discussion about how to handle pseudoscience postings. Santosh Patnaik writes: "Though Nature Precedings screens submissions for pseudo-scientific content, it is possible for such a submission to get through. An example might be this article on Nature Precedings: this website suggests that the authors are supporting “creationism/intelligent design.” This leads one to wonder how pseudo-science is identified, and what the policy is towards accepted articles that are later identified as pseudo-scientific."
Timo Hannay responds: "Nature Precedings does not accept pseudo-science. Spotting this and other inappropriate content is the job of our curators. In the month since launch they have already filtered out many such examples. In this particular case, the curator handling the contribution was concerned about the nature of the content and so consulted a senior journal editor with very considerable expertise in this area. In their opinion, it is not pseudo-science and is worth posting for community comment."
The discussion that follows touches on how to define and identify pseudoscience; whether preperint server managers should block postings of it; what tools can be used by the community to indicate quality; how an inchoherent writing style can obscure meaning; and the role of scientists in helping to promote "real" science and identify the flaws in pseudoscience, for the wider public.

Posted by Maxine Clarke on October 24, 2007
Categories: Quality and value, Technical solutions | Permalink | Comments (0)

Mark Chillingworth, the Editor of Information Week Review, writes in the October issue that there are debates on how to improve peer review, alludes to a recent PRISM statement about the need to protect it, but that "nowhere is there anyone laying out the true costs of peer review". He suggests that these costs need to be calculated as part of any informed way forward to maximize the benefits of the system.
Martin Blume, then Editor-in-Chief of the American Physical Society, wrote in the first of Nature's web debates on access to the literature in 2001:
"Peer review is expensive, and although reviewing by scientists is voluntary, we need to pay our editorial staff. It is more time consuming and hence more costly to review the 10,000 rejected articles than it is to review those that are accepted. Consideration is being given to other forms of peer review, but no savings are as yet obvious if quality is to be maintained."

Posted by Maxine Clarke on October 05, 2007
Categories: Quality and value | Permalink | Comments (0)

Dr Robert Higgs of the Independent Institute, writes:

Journalists, politicians and advocacy groups refer to “peer-reviewed research” and “scientific consensus” as the authoritative last words on controversial matters involving the natural sciences, from climate change to stem-cell research and genetically engineered foods. But many people have an unrealistic view of how the scientific community actually works.
The peer-review process is not, contrary to popular belief, a nearly flawless system of Olympian scrutiny. Any editor of a peer-reviewed journal who desires to reject or accept a submission can easily do so by choosing appropriate referees.
Unfortunately, personal vendettas, ideological conflicts, professional jealousies, methodological disagreements, sheer self-promotion and irresponsibility are as much part of the scientific world as any other. Peer review cannot ensure that research is correct in its procedures and conclusions. A part of the work in every discipline – from the physical sciences to economics –consists of correcting previous mistakes.
At any given time, “scientific consensus” may exist about various matters. Over time, however, new interpretations, tests or observations may demolish that consensus. For instance, in the mid-1970s, an apparent scientific consensus existed that our planet was about to enter another Ice Age. Drastic proposals, such as exploding hydrogen bombs over polar icecaps to melt them. and damming the Bering Strait to prevent icy waters from entering the Pacific, were put forth by reputable scientists and seriously considered by the US government.
The truth is that scientific research at the upper echelons occurs within a fairly small world. Leading researchers attend the same conferences, belong to the same societies, review one another’s work for funding organizations, and so forth. If you do not belong to this tight fraternity, it becomes extremely difficult to gain a hearing for your work, to publish in a “top” journal, to acquire a government grant, to receive an invitation to participate in a scientific conference, or even to place your grad students in decent positions.
“Scientific consensus” often emerges because the members of this exclusive club, and those who support them, have too much invested in the reigning ideas to let go. In this context, it behooves bright young scientists not to rock the boat by challenging anything fundamental or dear to the hearts of those who constitute review committees of funders or journals. The terms "peer review" and "scientific consensus" often serve to suggest a process of disinterested neutrality and saintly pursuit of truth. Like every other human endeavour, however, science is conducted by people with the full range of human emotions and motives.
Good rules of thumb for the non-scientist might be the following: government-funded research that is used to justify that government’s policy should be suspect, whether or not it’s peer-reviewed; and the research of scientists who appear at press conferences in the company of politicians or activists whose agendas they are there to support should be suspect, whether or not the work upholds the consensus opinion.

Robert Higgs is Senior Fellow in Political Economy at the Independent Institute, editor of the quarterly journal The Independent Review, and the author of Depression War and Cold War, as well as numerous books and more than 100 articles in scholarly journals.

Posted by Maxine Clarke on September 17, 2007
Categories: Quality and value, Systems | Permalink | Comments (5)

Pedro Beltrao of Public Rambling writes a stimulating post exploring whether the editor could be a "value unit" for authors and their publications, independent of the journal. He asks:"could there be freelance editors? Could the editors be separated from the publisher? Imagine [if] I read a paper from a pre-print server, ask some people to peer-review (why would they?) and sell our evaluation to a journal.
Also, can a publisher sell the editorial decision to another publisher? Lets imagine a journal that has a very high rejection rate, the editor asks referees for comments but ultimately the manuscript is rejected. The editor could then ask the authors where they want to send it next and offer to provide the referee report and editorial comments directly to the next journal to expedite the process. Could this journal get paid for this?"
Pedro is not referring here to the transfer system that some publishers, including Nature Publishing Group, offer between their journals, but to the editor taking a role rather like that of an agent in book publishing, as supervisor of the peer-review process and then advocate of the "package" of manuscript and reviews in finding the best possible journal in which to publish the work.

Posted by Maxine Clarke on July 24, 2007
Categories: Quality and value | Permalink | Comments (0)

The pseudonymous FemaleScienceProfessor is also an editor for a journal. She has reviewed its reviewers:

"I did a quick, statistically invalid analysis of the reviewer data for the past year to see whether the time it took a reviewer to complete the review was random or correlated with seniority. My working hypothesis was that younger scientists do quicker reviews. The dataset is sufficiently large to make an analysis like this reasonable, but I wasn't rigorous about tracking down reviewer time-from-Ph.D. data. I put reviewers in one of several bins: postdoc, assistant professor, mid-career, late-career, retired, and I put research scientists into these same bins based on where they would be in terms of time since Ph.D. if they were tenure-track. It's not a perfect system, but I just wanted to get a sense for any trends. The quickest reviewing groups are the early-career and retired scientists. "

More analysis is provided in FemaleScienceProfessor's post, and there are some reactions in the comments to this post. At the Nature journals we do not publish reviewer statistics of this type, nor do we capture information about the reviwers' gender, seniority level and so on. Is there interest from among our peer-reviewers to know these statistics, and to have published the type of information provided by the preliminary results of FSP?

Posted by Maxine Clarke on July 15, 2007
Categories: Quality and value | Permalink | Comments (0)

Among the excellent content on Nature Reports Stem Cells is an especially exciting regular feature, "Inside the Paper", which zeros in on recent scientific papers in the field published in Nature and provides detailed expert comment and author responses drawn from the peer-review process. The editors intend this feature to let readers see the paper’s context, strengths and caveats, as well as make the peer- review process a little more transparent.

Two Inside the Paper features have been published this month. The first, Cloning from Chromosomes, discusses the Article by D. Egli et al., Developmental reprogramming after chromosome transfer into mitotic mouse zygotes, in Nature 447, 679–685 (2007). Read a panel of experts' comments on this interesting article, responses from the authors, and comment yourself on The Niche (the Nature Reports Stem Cells blog).
The second Inside the Paper (21 June), Rewriting in blood, is about a paper by I. M. Samokhvalov et al(Nature 446, 1056–1061; 2007), reporting a surprising origin for blood stem cells. As before, you can read some of the peer reviewer's comments, responses from the authors and invite you to add your contribution on The Niche.

Posted by Maxine Clarke on June 28, 2007
Categories: Quality and value, Systems | Permalink | Comments (0)

The Journal of Electronic Publishing

The Influence of Academic Values on Scholarly Publication and Communication Practices.
by Diane Harley, Sarah Earl-Novell, Jennifer Arter, Shannon Lawrence and C. Judson King
This paper was refereed by the Journal of Electronic Publishing 's peer reviewers.
From the abstract: This study reports on five disciplinary case studies that explore academic value systems as they influence publishing behavior and attitudes of University of California, Berkeley faculty. The case studies are based on direct interviews with relevant stakeholders — faculty, advancement reviewers, librarians, and editors — in five fields: chemical engineering, anthropology, law and economics, English-language literature, and biostatistics. The results of the study strongly confirm the vital role of peer review in the choices faculty make regarding their publishing behavior. The perceptions and realities of the reward system keep faculty strongly adhered to conventional, high-stature print publications (and their electronic surrogates) as the means of reporting research and having it institutionally evaluated. Perceptions of electronic-only publications are frequently negative because those venues are considered to lack strong peer review and are, consequently, believed to be of relatively lower quality. There is much more experimentation, however, with regard to means of in-progress communication, where single means of publication and communication are not fixed so deeply in values and tradition as they are for final, archival publication. We conclude that approaches that try to "move" faculty and deeply embedded value systems directly toward new forms of archival, "final" publication are destined largely to failure in the short-term. From our perspective, a more promising route is to (1) examine the needs of scholarly researchers for both final and in-progress communications, and (2) determine how those needs are likely to influence future scenarios in a range of disciplinary areas.
The full article can be seen at the journal's website.

Posted by Maxine Clarke on June 13, 2007
Categories: Quality and value | Permalink | Comments (0)

The Niche: How can journals improve peer review of cloning papers?

This week we welcome a new Nature blog, The Niche, the blog of the (also new) Nature Reports Stem Cells website. In one of its first posts, Nature editor Natalie De Witt writes:

In the aftermath of the Hwang scandal in 2006, Nature editors thought long and hard about whether journals could employ editorial procedures that might prevent publication of such fraudulent data in the future, at least in the area of cloning and nuclear transfer research. We queried several top scientists in the cloning and stem cell fields on this issue, and published the major conclusions in the editorial entitled "Standards for papers on cloning" Nature 439, 243 (2006).
Several of these scientists have agreed for us to publish abridged versions of their 2006 answers in The Niche. Open the Comments below to read the postings of George Daley, Shin-Ichi Nishikawa, Alan Trounson, Alan Colman, Robert Lanza, Teruhiko Wakayama, Bob Wall, and Mark Westhusin, on whether the Hwang scandal could have been prevented, and what tactics journals should implement in the future to tighten up cloning papers. Feel free to join in the discussion by posting your own comments.
The questions:
1. Is Nature's current review procedure adequate as it stands for refereeing of cloning papers?
2. If not, what improvements are needed?
3. Would the peer review process of cloning papers be improved were Nature to establish a checklist of minimal standard criteria for authors and referees to refer to during the peer review process?
4. If yes, what sort of experimental data should be considered and to what extent do the raw data need to be presented to reviewers, and to what extent should it be published?
5. In 2006, Nature asked for independent verification of the Hwang paper showing that Snuppy is a cloned dog. Do you feel this kind of independent verification should be requested for all cloning papers? If yes, what should it entail?

You can read all the responses here, and add your own comments.

Posted by Maxine Clarke on June 07, 2007
Categories: Quality and value | Permalink | Comments (0)

Respectful Insolence: Cranks against peer review

The post linked to here is a long article about some recent "anti science" or "crank" attacks on the scientific peer-review system. It is a stimulating and informal read, concluding that the peer review system may have its faults, but that it is sensible to be wary of outspoken criticisims of it.

Posted by Maxine Clarke on May 18, 2007
Categories: Quality and value | Permalink | Comments (0)

Peer Review: A Necessary But Not Sufficient Condition

This is a link to an interesting post on the Real Climate blog about how peer-review aims to weed out poor science, whether it is successful at that goal. The examples given are from the field of climate research, and provide a telling account of how accuracy can be lost, advertently or inadvertently, in the glare of publicity. The post concludes: "even when it initially breaks down, the process of peer-review does usually work in the end. But sometimes it can take a while. Observers would thus be well advised to be extremely skeptical of any claims in the media or elsewhere of some new "bombshell" or "revolution" that has not yet been fully vetted by the scientific community."

Posted by Maxine Clarke on May 17, 2007
Categories: Quality and value | Permalink | Comments (0)

Research journal gives undergrads chance to shine

The article at the link above, in the McGill Reporter, highlights the McGill Science Undergraduate Research Journal, "a unique publication that offers undergraduate science students a rare opportunity to participate in the peer-review process, get their first publication credit and, in the end, improve their science. "

The article describes how the journal has just been relaunched, how the peer-review system works, and how it is funded and staffed.

Posted by Maxine Clarke on April 27, 2007
Categories: Journals, Quality and value | Permalink | Comments (0)

Shaping the Message, Distorting the Science - Center for Media and Democracy

Sheldon Rampton, Research Director of the Center for Media and Democracy, is testifying today (28 March) before the US House of Representatives Committee on Science and Technology in a hearing entitled "Shaping the Message, Distorting the Science: Media Strategies to Influence Science Policy." Mr Rampton's written testimony is available here, and a webcast of the hearing is available here. Mr Rampton discusses the reliability of scientific knowledge, and the way in which it has been, and is, used as a tool to manipulate public opinion. He provides some examples of cases in which industries (such as the tobacco and petroleum industries) have undermined the journal peer-review process, and concludes:

"The manipulation of science for public relations or political advantage inevitably has a corrupting effect on science itself. It undermines the integrity and objectivity of scientific research. It creates confusion in the minds of policymakers and the general public. What is needed, therefore, is greater public transparency regarding the sponsorship of science and of organizations that claim to speak on scientific matters. The public and policymakers have a right and to know who is funding research, what strings are attached to that funding, and how it may be affecting the information we use to make decisions—especially decisions on policy matters that affect us all."

Rampton has authored numerous articles, commentaries and books (with John Stauber) on the subject of this testimony ,including Trust Us We're Experts: How Industry Manipulates Science and Gambles With Your Future ; and Toxic Sludge Is Good For You: Lies, Damn Lies and the Public Relations Industry.

See here for the website of "Shaping the message, distorting the science": US House of Representatives.

Posted by Maxine Clarke on March 28, 2007
Categories: Ethics, Policy and Public, Quality and value | Permalink | Comments (0)

From the April editorial in Nature Chemical Biology:

"Although we strive for transparency in scientific publishing, there is an inherent need for opacity in the peer review process so that editors can receive candid information about the technical merit and potential impact of each manuscript. Given the limitation of referee confidentiality, the editorial team at Nature Chemical Biology is committed to making peer review as transparent as possible. Referee selection is a critical element in this process. We request that potential referees notify us of possible conflicts of interest that may affect their ability to provide an objective assessment of a manuscript. When potential conflicts arise, referees may recuse themselves. Conflicts may include competing financial interests related to the science reported in the article or personal or professional relationships with authors that may affect the referee's impartiality. To minimize conflicts in which scientists may be in competition or hold opposing views on a scientific controversy, the editorial team generally honors author requests to exclude particular scientists from the referee pool. Finally, to enhance transparency, Nature Chemical Biology editors communicate with referees after a decision has been sent to authors. In addition to informing referees of our editorial decision, we send copies of all referee comments, while maintaining the confidentiality of all referees, to each reviewer who was consulted on the manuscript."

The complete editorial is at: Nature Chemical Biology 3, 185 (2007).
Nature journals' policies on competing financial interests are explained at the author and referees' website.

Posted by Maxine Clarke on March 20, 2007
Categories: Quality and value | Permalink | Comments (1)

In peer review, not all peers are equal: so writes John Timmer on ars technica, reporting on a limited study of the quality of peer-review on the journal Annals of Emergency Medicine. Although the study was limited to one journal and hence discipline, the dataset consisted of 306 reviewers who had completed more than 3,000 reviews.
"Satisfactory reviews were compared to unsatisfactory ones, and excellent reviews were separately compared to average. For both of these cases, correlations were tested with a number of factors thought to improve reviewing skills (participation in grant reviews, years of experience, prior coursework in critical appraisal, etc.). These factors were tested both individually and in a multivariable analysis, which should eliminate any confounding factors.
In news that may be disturbing for journal editors everywhere, very few factors leapt out as having a consistent and significant correlation with the quality of a review, although some factors did have strong correlations in individual tests. The only positive factors linked to quality of reviews were age (younger reviewers were better) and working at an academic hospital. Ironically, service on an Institutional Review Board, which evaluates and approves experiments on humans, consistently correlated with lower-quality peer reviews. Even these factors, however, were only slightly better than random at predicting review quality."
Dr Timmer concludes: "There are some clear limitations to this study, given that it applies to a single journal with content that's exclusively medical. But the researchers who performed it note that it may be difficult to extend these studies to other fields, as many journals don't even have a mechanism for evaluating or tracking the quality of reviewers. Given the importance of peer review to the entire scientific enterprise, their strongest conclusion is that more needs to be done to track and evaluate the process in order to ensure that the body of published information is as reliable as possible."

Posted by Maxine Clarke on March 09, 2007
Categories: Quality and value | Permalink | Comments (0)

Doug Lederman reports in Inside Higher Ed on competition in US agricultural research. "The notion that competition produces better science is an underlying principle of the American research enterprise — with the possible exception, historically, of agriculture. Just 10 percent of the more than $2.6 billion that the federal government spends annually on agriculture research is awarded through peer review, with the rest flowing to scientists at the U.S. Agriculture Department or to researchers at land-grant colleges through formulas that have gone largely unchanged for decades."
Mr Lederman describes the current Senate hearings, and various attempts, including that of the Create-21 group, to increase the proportion of funding given to competitive research programmes.

Posted by Maxine Clarke on March 09, 2007
Categories: Quality and value | Permalink | Comments (0)

At Nautilus, the blog for authors, I've posted about competing interest statements in scientific papers. I've asked there what scientists think of this practice as authors. But what about when the scientists are peer-reviewers? Does it make a difference to you, when you are reviewing a paper, to know that the author has taken out a patent on the discovery or has shares in a spinoff company? Do peer-reviewers judge the scientific results independently of these declarations, or does it make a difference to the level of scrutiny they apply to the work? The Nature journal editors would like to hear about the reviewers' perspective, via your comments to this post.

Posted by Maxine Clarke on December 19, 2006
Categories: Ethics, Quality and value | Permalink | Comments (0)

Tom Jefferson

A look at alternatives to the peer review system suggests it might be the best bet.

Peer review is used as a quality-control mechanism for biomedical literature (ref. 1). Journals with a formal peer-review system are generally higher profile than those without, and peer review is often a pre-condition for journal indexing in biomedical databases such as PubMed. Variants of the current system or complete alternatives have scarcely been explored, as yet. Two years ago, I and my colleagues carried out an exploratory review of complete and partial alternatives to the current editorial peer-review system, both in and outside biomedical sciences. Here I update and summarise our findings.

Continue reading "Quality and value: Models of quality control for scientific research" »

Posted by Maxine Clarke on June 29, 2006
Categories: Peer review debate, Quality and value | Permalink | Comments (1)

Charles G. Jennings

What you can’t measure, you can’t manage: the need for quantitative indicators in peer review

Given its importance in steering the global research enterprise, peer review seems under-studied. There is a growing literature on the subject, some of which is highlighted at the quadrennial Peer Review Congress, but for the most part we are still only seeing snapshots. A more systematic approach is needed if we are to understand peer review as it is currently practiced, or to evaluate the pros and cons of any alternative approaches.

Continue reading "Quality and value: The true purpose of peer review?" »

Posted by Maxine Clarke on June 29, 2006
Categories: Peer review debate, Quality and value | Permalink | Comments (0)

Trish Groves

A recipe for good peer review

Improving peer review depends on making its human aspects more humane. Journals need to ask the right reviewers to review the right articles, help them to do it quickly and thoroughly, make them feel happy to sign their reports, thank them, tell them how they did, and encourage wide recognition of what’s too often a thankless task.

Continue reading "Quality and value: How can we get the best out of peer review?" »

Posted by Maxine Clarke on June 20, 2006
Categories: Peer review debate, Quality and value | Permalink | Comments (0)

David Ozonoff

Researchers need reviewers to check their stats.

Statistical methods are widely used in many areas of natural science, especially in my field of research, epidemiology. Although statistical procedures are often viewed as a black art, or as a black box, they are not limited to specialists. With today’s computing power and software, researchers can and do use computationally intensive methods of great complexity, often leading to the use of techniques that are more sophisticated and powerful than necessary. Many researchers have trouble interpreting the results, or interpret them incorrectly. Clearly, this is a matter for peer review.

Continue reading "Quality and value: Statistics in peer review" »

Posted by Maxine Clarke on June 14, 2006
Categories: Peer review debate, Quality and value | Permalink | Comments (4)

Joan E. Sieber

Improving the peer-review process relies on understanding its context and culture.

Peer review gets a bad rap these days, and there is much talk that something should be done about it. Scientific societies need to know more about the alleged problems before they try to fix the peer-review process. These problems are often cast as ethical ones: unintended bad consequences of well-meaning editorial processes.

Continue reading "Quality and value: How can we research peer review?" »

Posted by Maxine Clarke on June 05, 2006
Categories: Peer review debate, Quality and value | Permalink | Comments (3) | TrackBacks (2)