Peer-to-peer

The Editorial 'Working double-blind' (Nature 451, 605–606; 2008), also republished on this blog and stimulating more than 70 comments, referred to a study (1) that found more female first-author papers were published using a double-blind, rather than a single-blind, peer-review system. The data reported in ref. 1 have now been re-examined (2). The conclusion of ref. 1, that Behavioral Ecology published more papers with female first authors after switching to a double-blind peer-review system, is not in dispute. However, ref. 2 reports that other similar ecology journals that have single-blind peer-review systems also increased in female first-author papers over the same time period. After re-examining the analyses, Nature has concluded that ref. 1 can no longer be said to offer compelling evidence of a role for gender bias in single-blind peer review. In addition, upon closer examination of the papers listed in PubMed on gender bias and peer review, we cannot find other strong studies that support this claim. Thus, we no longer stand by the statement in the fourth paragraph of the Editorial, that double-blind peer review reduces bias against authors with female first names.
References
1. Budden, A. E. et al . Trends Ecol. Evol. 23, 4–6 (2008).
2. Webb, T. J. , O'Hara, B. & Freckleton, R. P. Trends Ecol. Evol. doi:10.1016/j.tree.2008.03.003 (2008).

Posted by Maxine Clarke on June 05, 2008
Categories: Ethics, Quality and value, Systems | Permalink | Comments (2)

Ignazio R. Marino* writes in Correspondence in the current issue of Nature (453, 449; 22 May 2008):
'Italy must invest more in science and technology' according to I. Bertini, S. Garattini and R. Rappuoli in Correspondence (Nature 452, 685; 2008). They lament the Italian lack of financial resources and political attention for research, technology and education. As a researcher, clinician and academician, I share their concerns. However, as former chair of the health committee of the Italian Senate, I take exception to their implication that none of the major political parties recognizes science, technology and education as crucial for the future of the country's economy.
The 2007 and 2008 national budget laws, drawn up when the centre-left coalition was in power, allocated 96 million (US$149 million) to projects submitted by researchers under 40 years old. These are judged by an international committee comprising ten scientists under 40 — five from foreign institutions — selected according to impact factor and citation index scores. This alone is a revolutionary approach for the unregulated Italian system of research funding allocation.
In spite of such advances, Italy is still far behind in research investment, and this needs to change. But the crucial switch is not simply to increase funding. The way the new government should proceed is to reform the allocation criteria for funding and to start applying across the board the selection and evaluation rules of peer review. Such a system would acknowledge meritocracy and free researchers from the virtual slavery under which they have been kept by old academicians.
By applying international rules of peer review and evaluating grant applications only on the basis of merit, looking at curricula and objectives, comparing lists of publications and evaluating results, we will provide opportunities for Italy's scientists, thereby promoting the country's intellectual, cultural and economic growth.
*Department of Surgery, Jefferson Medical College, 19107 Philadelphia, Pennsylvania, USA, and Senate of the Republic of Italy, Piazza Madama snc, 00186 Rome, Italy.

Posted by Maxine Clarke on May 27, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)

From an Editorial (free to access online) in last week's Nature (453, 258; 15 May 2008):

At first glance it seems to be a shining example of the scientific method in action. Two papers published by biochemist Homme Hellinga and his students at Duke University Medical Center in Durham, North Carolina, claimed a breakthrough in rational enzyme design. Last year, another chemist found that Hellinga's enzymes didn't actually work, which led to the retraction of the two papers this February (see page 275 of this issue and an earlier story on 13 February). Then, this March, a third group published research showing that rational enzyme design really is possible. All has ended happily, it seems, with the field marching forward in triumph.
But examined more closely, the episode reveals some less than happy aspects of science as it is actually practised. For example, the problems with Hellinga's enzymes were identified by John Richard at the State University of New York in Buffalo, who hoped to use the proteins in his own work. In effect, Richard and his two co-workers wasted seven months and tens of thousands of dollars failing to reproduce the results from Hellinga's lab. Richard's subsequent efforts to correct the scientific record thus came at considerable cost, with no discernable benefit to his own career.
This is a perennial problem in science. Many researchers who come across non-reproducible work save themselves extra hassle and money by simply not pursuing it further. Meanwhile, those who refuse to let it go — like Richard — gain nothing.

Continue reading "Hellinga story exemplifies weaknesses of the scientific process" »

Posted by Maxine Clarke on May 21, 2008
Categories: Quality and value | Permalink | Comments (0)

This post is a continuation of the discussion about blogging and peer-review by selected reactions at RealClimate (a climate scientists' community blog) to Nature Geoscience's two commentaries on blogging.

Philip Machanick writes that the "problem with blogs is that there is no way for an outsider to know which are reasonably careful creations of informed scientists, which are opinions of the scientifically illiterate, and which are astroturf creations designed to confuse critics of science that is in conflict with an industry. Given that terrain, I would rather have something like RealClimate than not: it helps to balance things out. Errors tend to be corrected quickly here as a consequence of a large informed readership (even if it is sometimes annoying that you get drive-by ignoramuses who don’t benefit from getting their misconceptions answered). He goes on to suggest that a site such as RealClimate is of value in providing a forum for rebuttals of peer-reviewed science in order to develop a consensus on whether a "formally published rebuttal is worth the effort".

Ray Ladbury's view: "When it comes to peer reviewed papers, one has to presume the reader will have a minimum level of familiarity with the subject matter. One also presumes that the reader will have a day job, and so the question becomes whether the information in the paper is of sufficient interest to the average scientist in the community to say, “Hey, take a look at this. It looks mostly correct to me and has some interesting information/insights/methods…” This is not in any way the gold standard in the sciences. The gold standard comes when the community as a whole says, “Hey, cool, I can use this.” The paper is cited. The techniques are used. Science advances. Eventually, what was in the paper becomes part of the tacit knowledge assumed by reviewers.
The tacit knowledge one can presume for a blog like Realclimate is much lower. One presumes there is an interest in the subject–why else would the reader be perusing the blog. One presumes at least a passing acquaintance with the scientific method and maybe some familiarity with basic results like conservation of energy, etc. One could perhaps assume that the average reader has taken the time to acquaint him- or herself with material to which one is vectored via the “Start Here” button–although this is far from Universal.
For the average newspaper reader of a science story, the tacit knowledge is nearly nonexistent–or worse, wrong. And then we have the blogosphere, where information density is at best, rarified and often toxic.......in an information economy, it seems that all too many readers and journalists are content to remain paupers."

Posted by Maxine Clarke on April 29, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)

In a comment to an earlier Peer to Peer post, 'Regular Scientist' takes issue with Nature Cell Biology's definition of a good report, writing (slightly edited for length): the journal sometimes asks the authors to undertake many experiments even before sending a paper to review. Not strangely, papers in Nat Cell Biology, even small reports, contain many Supplementary Figures (up to 20, I´ve seen). I miss the old days where publications were for sending a very interesting result so you could extend it and discuss its importance for your field. Nat Cell Biology's system is undermining research efforts in the authors´ labs, which have to dedicate an enormous effort to provide many additional experiments that don´t even add much to the concept. This is not what science, and the peer-review process, should be about. I would recommend the editors of Nat Cell Biol a calm reading of the original paper on the structure of nucleic acids, and note the way it was written.

Bernd Pulverer, Chief Editor of Nature Cell Biology, has kindly agreed to my request to write a guest post to respond, and to elaborate on the journal's practices and policies:

As ‘Regular scientist’ points out, we do strive to publish rather well-developed studies in Nature Cell Biology. It is interesting to read the concern about the supplementary information, since we usually hear the opposite complaint: namely that the tight format restrictions applied preclude presentation of complete datasets. In considering submissions, we expect fewer details of molecular mechanistic analysis if the paper reports a highly novel observation. And because the study of cell biology tends to throw up surprising new findings that deserve a wide audience but that cannot be developed in a reasonable timeframe, we decided some years ago to publish a short ‘Brief Communication’ format, which I hope is appreciated by Regular Scientist.
In the Editorial under discussion, we wrote:

Nevertheless, the most important part of the report is assessment of the data: are key experiments or crucial controls missing? Are the data significant and definitive? Are all claims made supported by the data? A black and white model should not be a requirement — in fact it often underestimates biological complexity. Formulate a clear set of recommendations for additions or improvements. Experimental suggestions are important even if an outright rejection is recommended, as it makes for a transparent and constructive report that will allow the author to improve the dataset and select an appropriate target journal. If the overall assessment is positive, do not feel obliged to ask for non-essential experiments. New experiments should add key information, buttress claims or improve existing data, and they must be achievable. Indeed, the perception that referees invariably ask for more data encourages some authors to hold back data in the hope that these will be requested and can then be provided by return post; clearly, this is a futile exercise. Referees and editors alike should take great care that all key issues are raised in the first assessment — draw a line and adhere to it. Subsequent evaluations may raise issues on new data, but raising yet more ways to develop a study is not fair.

We, the Nature Cell Biology editors, agree that one can always ask for more data, and that sometimes data require an unrealistic amount of effort for the information added. However, we entirely disagree that our editorial and review process does not result in the publication of much-improved datasets. The tendency of referees to feel an obligation to ask for some new data, and conversely for authors to hold back data to give referees something obvious to ask for, is an issue that we tried to address in the Editorial.
We certainly do not ask for more data because we, the editors, feel under any compulsion to do so: the simple fact is that a good number of manuscripts submitted to the journal are rather premature, and we cannot justify wasting our referee’s time when we perceive obvious holes in a dataset that the referees would invevitably want to be addressed. We believe that this step is in an author’s interest, since we will not undertake multiple rounds of peer-review without good reason. If requested data are deemed to be unobtainable for technical reasons, or indeed unnecessary, we are always willing to hear an author’s arguments.
Regarding the 1953 Nature paper that Regular Scientist recommends: indeed, the paper is a good read. However, science has become much more complex since then, and the discoveries often more detail-oriented, requiring a more in-depth analysis. At the same time, the tools available have developed tremendously in 55 years. Even in 1953, readers will note that Watson and Crick's conclusions depended on the two following papers in the issue, providing experimental details. The challenge is to set ‘the bar’ correctly to what is achievable and what should be achieved for publication in a top-flight journal. This is what the job of an editor is all about.

Posted by Maxine Clarke on April 23, 2008
Categories: Quality and value | Permalink | Comments (1)

Continuing the discussion of the relative contributions of blogs and the peer-reviewed literature to scientific understanding, I'm highlighting another reaction to the two Nature Geoscience Commentaries presenting different perspectives.
In the comment thread at Real Climate blog, Simon Donner of the University of British Columbia writes:

"Blogs have created a forum for many people to informally discuss science. They are also a great forum for scientists to provide context to their own work and to the work of others in their field.
And some of the time, that may take the form of criticism. That’s ok. But we do have to be careful in blurring the line too much between peer-reviewed publications and blogs. The first problem is the obvious one. Blog posts are unfiltered, un-reviewed, and often written off the cuff, while journal articles are screened, reviewed, and (should be) meticulously researched. It is far easier to write a criticism of a paper on one’s blog than to write a response and submit it to the journal. The issue isn’t just that no reviewers check the work… the blog author is unlikely to do even close to the amount of research and analysis nor give the wording nearly the same level of consideration as is expected in a paper submitted to a high-quality, peer-reviewed journal.
The other problem is that blog posts are readily accessible to anyone at anytime, both in language, and in the unlicensed nature of the internet. To a huge swath of the public, blog posts are the face of science. Like it or not, bloggers with scientific credentials are like self-appointed ambassadors for science. If we are going to write about our science, we should do it with thought, and we should do it well. That is a standard that myself and many other science bloggers often struggle to meet."

Posted by Maxine Clarke on April 16, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)

The two Nature Geoscience Commentaries expressing opposing views on blogging's role in science research communication have been much discussed in the blogosphere (see Climate Feedback blog, for example.). One discussion took place at RealClimate blog, where Gavin Schmidt's post about his Nature Geoscience view on how science blogs and traditional peer-review intersect has attracted more than 100 online comments from climate scientists and others.
One commenter, "Tamino", writes: "Blogs don’t serve very well for communication among scientists. Peer review does more than just protect us from being inundated with substandard work; it protects authors from their own mistakes and improves the quality of what we write. Peer review itself is an immensely valuable avenue of communication; who among us hasn’t at some time included a phrase like “We thank an anonymous referee for comments and suggestions which dramatically improved the final manuscript”?
But as bad as blogs are for actual research, peer-reviewed journals are far worse for communicating with and educating the lay reader. Yet when it comes to climate science the lay public is hungry for knowledge, and many of them are eager, and well-prepared, for a level of sophistication and detail that can’t be found in lay journalism or even popular literature; An Inconvenient Truth isn’t enough. So blogs serve an incredibly useful purpose, enabling the interested and well-educated reader to share insights with researchers who are at the cutting edge of new knowledge.....................while blogs aren’t part of the machinery for legitimate scientific research, they’re an indispensible tool for communication and combating misinformation."

More to follow.

Posted by Maxine Clarke on April 10, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)

Nature Cell Biology continues its coverage of peer-review in its (free access) April Editorial 'Good review' (Nat. Cell Biol. 10, 371; 2008).
After stating the rationale for peer-review, the Editorial asks whether reviewing is essentially an altruistic activity, or whether there are significant rewards for doing it. "The biggest benefit of refereeing is probably to stay engaged with the scientific community in a sort of 'real-impact journal club'. We review less than a quarter of submitted manuscripts, with the aim of involving referees only for studies that seem worthy of publication. This saves time for authors, and referees know that the manuscripts they receive meet a minimum standard, so that refereeing should be an enriching experience. Nevertheless, the time commitment for refereeing is considerable and we are campaigning to ensure that this is taken into account in assessing research performance."
Although much has been said about alternatives and enhancements to 'single-blind' peer review, what actually constitutes a good referee report? The rest of the Editorial provides some answers, which are further explored at our author and peer-reviewers' website.
Nature Cell Biology's previous editorial (March 2008) is discussed here.

Posted by Maxine Clarke on April 04, 2008
Categories: Quality and value | Permalink | Comments (2)

Scientists know much more about their field than is ever published in peer-reviewed journals. Blogs can be a good medium with which to disseminate this tacit knowledge, according to Gavin Schmidt of the NASA Goddard Institute of Space Studies Columbia University, and co-founder of RealClimate.org, in a Commentary in the April issue of Nature Geoscience (1, 208; 2008). Dr Schmidt asks "why read a blog when you can go directly to the scientific literature? Unfortunately, access to new findings in the traditional way is harder than it should be. Many technical papers are behind pay-walls, which make it impractical and expensive for unaffiliated individuals to read them. More importantly, however, even when papers are freely available, they often do not provide the insight expected."
In another Commentary in the same issue of the journal (Nat. Geosci. 1, 209; 2008), Myles Allen of the University of Oxford agrees that "explaining science to journalists and the public on blogs is fast and efficient." But after his own bruising experience, he wonders: "is it all just too good to be true? Can science survive Web 2.0?"
After describing his own difficult experiences of having his own peer-reviewed work criticised and misinterpreted by people who would not subject their own conclusions to peer-review, Dr Allen concludes: "Everyone agrees we need to communicate science better to the general public. But more and faster should not be confused with better. I'm certainly not advocating closing science blogs or discouraging science websites. We just need to remember the basic courtesies that our doctoral supervisors took for granted: criticism of peer-reviewed results belongs in the peer-reviewed literature. Direct communication over the Internet, far from creating a level playing field, just ploughs it up and makes the game impossible. The problem is, without witty and cutting criticisms, what is the point of a blog? Sure, the peer-review system is creaking. Sure, science journalism sometimes trips up. But like Churchill's quip about democracy, it is the worst possible system for communicating scientific results, apart from all the alternatives that have been tried from time to time."
Yet Dr Schmidt remains optimistic: "Some may dismiss blogs as being a distraction from real scientific work, or of egging on the very controversies that we seek to diffuse. There is an element of truth to both of these claims. But the response should not be a return to the ivory tower. That simply leaves the field clear for those who prefer to confuse rather than enlighten. With the importance of science in policy decisions being more apparent than ever, our ability to do science and enhance its relevance in public life relies on the community's willingness to engage, inspire and inform. Blogs are one way to do that, and they can excel at providing the context that is so often missing in other media. Not every scientist needs to have one, but maybe every scientific field does."

Posted by Maxine Clarke on April 01, 2008
Categories: Policy and Public, Quality and value | Permalink | Comments (0)

Nature Neuroscience is joining a consortium of journals that enables reviews to be transferred from one journal to another, while allowing authors, referees and editors to control their degree of participation in the system flexibly. The reasons for the decision are explained in this month's (April) Editorial (Nat. Neurosci. 11, 375; 2008). Briefly, in January, a group of editors, supported by the Society for Neuroscience, implemented a system for transfer of submitted manuscripts between journals that allows voluntary participation by authors, referees and editors, known as the Neuroscience Peer Review Consortium. This consortium reduces the overall reviewing workload of the community by allowing authors to continue the initial review process when their paper moves from one consortium journal to another, once the paper has been rejected or withdrawn from the first journal. This arrangement is similar to the manuscript transfer system that has been available within the Nature family of journals , and all the other journals published by Nature Publishing Group, for almost a decade.
The neuroscience transfer system, described in detail in the Editorial and at the Neuroscience Peer Review Consortium website, is voluntary for authors and peer-reviewers. Journal editors have full discretion in deciding how to use transferred reviews. The receiving editor may choose to accept or reject a paper based on these reviews, without further consideration; to send the paper to some or all of the previous referees for evaluation of the authors' revisions; or to request a fresh set of reviews from new referees. Only comments to the authors are transferred to the receiving journal. Confidential comments to the editors are not passed along. Thus, to ensure transparency in the review process, both at Nature Neuroscience and at other journals after the paper has been transferred, journal editors encourage referees to include all their concerns about the paper in comments to the authors. According to the Nature Neuroscience Editorial, the small amount of extra time required to phrase comments diplomatically for the authors should be more than counterbalanced by the resulting improvement in the peer-review process.
Many members of the community have strong views on the issue of confidential comments, which can be found on the Action Potential blog, and which has previously been discussed at Peer to Peer. The success of the neuroscience journals' transfer system will be evaluated at the end of this year. Nature Neuroscience editors will also be evaluating the journal's participation on an ongoing basis, so they encourage authors, referees and readers to share their comments with the editors, either on Action Potential blog or privately by e-mail.

Posted by Maxine Clarke on March 31, 2008
Categories: Ethics, Quality and value, Systems | Permalink | Comments (0)

How many papers does the typical researcher review in a year? How long do they take? And why do they do it? For the answers, read the Editorial in this month's (March) issue of Nature Nanotechnology, "Who'd be a referee?" (3, 119; 2008). The Editorial reports some of the findings of the recent Publishing Research Consortium report on peer-review (previously discussed at Peer to Peer), whose survey revealed that the "average review takes about 8.6 hours (with a median of about 5 hours) and is completed within 3–4 weeks, although there are significant differences between the four broad subject areas covered by the survey: in the physical sciences and engineering, for instance, the average (mean) is 10.4 hours, compared with 6.3 for clinical researchers. Of course, on top of this, many researchers spend considerable time reviewing grant applications for funding agencies, which can be equally onerous and possibly even more important than reviewing papers."
For similar statistics, and a more general discussion of the peer-review process in publishing, see the Nature Nanotechnology Editorial, or see here for Nature Cell Biology's take.

Posted by Maxine Clarke on March 10, 2008
Categories: Quality and value, Systems | Permalink | Comments (0)

Much attention is being given to a Commentary in the current issue of Nature (Nature 451, 379-399; 2008), A tale of two citations, some of which I have attempted to encapsulate in this Nautilus post, for those interested. Although the issues immediately concern a possible increase in duplicate publication and plagiarism, as detected by software systems and database searches, peer reviewers are an integral part of the check/balance procedures that journals use. For this reason, I thought it well worth highlighting here the comment by Brian Derby at the Nature Network forum currently discussing these questions. Part of Dr Derby's response:

"As a referee I have identified duplicate or severely overlapping content while reviewing papers in the past (for reasonably high profile/impact factor journals). I do not search for duplication routinely but, as someone who is used to referee papers in particular niche areas, I received both papers in one instance and in another I had read an on-line pre-pub before receiving the duplicate. The authors will not be named as that would break referee confidentiality but they were from well known institutions in the developed world.
What was the common factor (apart from the paper!) – the authors were relatively junior new appointments. Younger academics seem to feel themselves under a lot of pressure to publish. In my department I believe that my younger colleagues are much more sensitive to impact factor than is possibly healthy when they consider where to publish an article."

Posted by Maxine Clarke on January 28, 2008
Categories: Ethics, Quality and value, Systems | Permalink | Comments (1)

The Publishing Research Consortium publishes a study this month (January 2008) in whch more than 3,000 senior authors, reviewers and editors were asked about the peer-review system. The conclusions are that researchers want to "improve, not change, the system of peer review for journal articles". According to the report, a summary of which is available (1.7 MB; PDF), more than 93 per cent of respondents believe that peer review is necessary, and more than 85 per cent say that it helps to improve scientific communications and increases the overall quality of published papers.
Although many respondents pointed out the operational difficulties in double-blind peer review, two-thirds of respondents felt that it is the most objectively fair system, compared with single-blind (the current prevalent system). Alternatives such as post-publication and open peer-review were not popular.
While of the majority of respondents saw peer review as an effective filter for research, some did not think it was effective at detecting plagiarism, fraud or misconduct. Interestingly, most reviewers among the respondents thought that paying peer-reviewers would be too expensive for publishers; most of them said that they perform reviewing as part of their support to their research community.
The full report is available here (1 GB; PDF). According to the Publishing Research Consortium, the main objective of the study was "to measure the attitudes and behaviour of the academic community with regard to peer review. This will inform debate concerning peer review, and underpin discussions, either in discussion lists or at future workshops/conferences."
This new report comes as the NIH (National Institutes of Health) finish analysing the thousands of responses to their assessment of grant peer review. Lawrence Tabak and colleagues are filtering the list into a set of key recommendations, which will be given to Elias Zerhouni, director of NIH, at the end of February.
Update, 29 Jan 2008. Nature Neuroscience discusses the NIH peer-review exercise in its February issue Editorial (Rethinking grant review Nature Neuroscience 11, 119; 2008).

Posted by Maxine Clarke on January 25, 2008
Categories: Quality and value, Systems | Permalink | Comments (0)

In an interesting exercise in how a journal can develop criteria for peer-review in a fast-moving, area of considerable intrinsic uncertaintly, the editors of Nature Reports Stem Cells posted an article last month that asked how one could declare human cells pluripotent, when the most robust tests are neither ethical nor feasible, calling on the US National Institutes of Health (NIH) to set standards. Now, in a post on The Niche blog, Defining pluripotency in human cells, the editors post some of the responses from researchers in the field. Here is one perspective, from Shinya Yamanaka of Kyoto University:

"This is an important, but difficult question. First of all, we don't know whether human ES cells are really ES cells or not. Because the lack of chimera experiments, we will not be able to answer this question. This means we lack a positive control. I have been telling my students that one of the worst experiments you can do is one without positive and negative controls."

The Niche also features an interview with Story Landis, head of the stem cell task force at NIH, about the world's response to the recent breakthrough result that human cells can be reprogrammed, including NIH's problem about how to develop criteria for funding in a research area that in itself does not support the US President's policy.

I will not be posting again on Peer to Peer until the New Year, so I wish you all a very happy Christmas and holiday season. If you feel the need for some interactive scientific discourse during this period of traditional peace for those involved in journal production, I can recommend Nature News (you can comment online on the articles) and Nature Network, where there are a range of groups, blogs and forums to suit any science-related interest, and to which you are welcome to contribute.

Posted by Maxine Clarke on December 19, 2007
Categories: Quality and value | Permalink | Comments (0)

In his Nature Network summary of a recent publishing meeting, Yiorgos Apidianakis describes his opinion that a peer-review "score" is a more desirable (and efficient) indicator of scientific excellence than the currently used impact factor of the journal that publishes the work. Charles G. Jennings's wrote, as part of Nature's peer-review debate, “It is common to bemoan the over-reliance on quantitative markers such as impact factors for assessing scientists’ abilities (and indeed there is much to bemoan), but until committee members have time to read every paper on every applicant’s CV, they will have to rely at least in part on proxy indicators.” Dr Apidianakis believes that an ideal indicator would be a score from a unified peer-reviewing system, or a central agency "that will thoroughly, rigorously and objectively evaluate any given work to be published, using again specialized scientists as reviewers. Having an evaluation score from such an agency, scientists can include this score to their publication record."

Dr Apidianakis thinks that such a system could work in practice, using the example of the NIH evaluation scoring system -- while admitting it would be very expensive. Irrespective of this practical obstacle (as well as others) I wonder how the quality of the agency's "score" can be standardized? The establishment of, in effect, one giant journal of research, with all the world's scientists signed up to it as peer-reviewers, is a stimulating concept. But, given the many divergences of views within fields, how could a centralized scoring system work? The IPCC has attempted a similar kind of approach for one discipline, climate change -- even though most scientists in the field broadly agree with the IPCC's assessment of research output, this consensus requires massive bureaucratic baggage, including many international meetings and vast reports justifying decisions. Yet there is a substantial minority of scientific dissentors, and many members of the public are sceptical of a unified approach resulting in "science by endorsement".

The current peer-review system works very well "in miniature", whether as operated by those journals able to call on the most thoughtful scientists in a field who do the most cutting edge research, or by literature reviews or research ranking services, usually written or operated by one or a few individuals. Scientific research itself has benefited from large author collaborations across the whole spectrum of disciplines, from astronomy and nuclear physics to genomics and cell signalling, facing the challenge of making sense of vast quantities of data. Are there the same intellectual and innovative advantages to be gained by a single-managed peer-review system for all of science?

Posted by Maxine Clarke on December 04, 2007
Categories: Quality and value, Systems | Permalink | Comments (0)

In a blaze of colour on the 11 November 'op-ed' (invited opinion) page of The New York Times, some scientists proclaimed that, based on analysis of brain-imaging data from just a handful of swing voters, they had divined what the rest of the undecided masses truly think about the upcoming US presidential elections. Apparently just asking them was simply not good enough.

So opens an Editorial in the current issue of Nature (450, 457; 2007), which goes on to describe how the authors of the New York Times piece used functional magnetic resonance imaging (fMRI) to scan the subjects' brains while looking at pictures of candidates. The Nature Editorial goes on:

"The op-ed work has not been published in a peer-reviewed journal, and the article is self-evidently too insubstantial in scientific detail to assess the strength of either the methods or the data. A group of cognitive neuroscientists was swift to object to its conclusions — which veer close to a modern-day phrenology — in a response to The New York Times.
The results described in the op-ed are apparently the claims of a commercial product posing as a scientific study. This is only partially transparent. Three of the authors list their affiliation with FKF Applied Research, a company based in Washington DC that is notorious for using similar brain-scan analysis to conclude which TV adverts aired during a major sporting event were most effective. In its own words, the company is a "business intelligence firm selling fMRI brain scan-based research to Fortune 500 companies".
Articles on The New York Times op-ed pages are opinionated by definition, and shouldn't normally require peer review. But here, the paper's editors have instead published the results of (to put it mildly) questionable scientific research, disseminating this information to millions of their readers who may not have the background to recognize for themselves the absurdity of some of the authors' conclusions.
Although it is a gross disservice to science and indeed to politics, it is a great deal for the company. Scientific publication would have required the authors to divulge their data and qualify their assumptions — and some journals might even have required that they declare their financial interests. Whatever the motives, seducing The New York Times' editors with the allure of Technicolor brains lighting up with Hillary Clinton angst yielded no more or less than a multimedia advertisement for the company's product to millions of readers."

Posted by Maxine Clarke on November 26, 2007
Categories: Ethics, Policy and Public, Quality and value | Permalink | Comments (0)

In an Editorial in today's Nature (450, 457-458; 2007), Replicator review, the journal describes its publication of "what we expect to be the final word on whether nuclear transfer can work in a primate — a paper by J. A. Byrne et al. showing not only that it is possible to clone primate embryos by somatic-cell nuclear transfer but also that precious embryonic stem cells can be derived from the embryos. If embryonic stem cells live up to their promise, the technology could be used to derive patient-tailored stem cells."
The Editorial goes on to describe how Nature took the unusual step of soliciting an independent verification of the paper during the process of peer review. "This is the first time that Nature has obtained second-party replication ahead of publication. It should not be seen as reflecting a mistrust of scientists in the cloning field or scepticism about this particular research group. Rather, our actions fulfilled a statement in an Editorial from 2006 that was conceived in the aftermath of the Hwang affair: "Keeping in mind the principle that extraordinary claims require extraordinary proof, Nature may in rare cases demand it."
Nature will continue to evaluate the need for such validation on a case-by-case basis. Meanwhile, researchers should consider maintaining tissue samples, and trying to establish and fulfil requirements that may go beyond the routine, as Byrne et al. did for their cloning paper. In that way, remarkable scientific developments can remain as free as possible from unwarranted speculation and controversy."

Also in this issue of the journal is a News and Views article on the new work, by Ian Wilmut and Jane Taylor (Nature 450, 485-486; 2007); a News story featuring an interview with one of the authors of the paper; and a Nature editors' one-paragraph summary of the work.

A blog discussion of peer-review challenges in stem-cell research and the peer-reviewers' reports for the paper by Byrne et al. can be found at Nature Reports Stem Cells.

Posted by Maxine Clarke on November 22, 2007
Categories: Quality and value | Permalink | Comments (0)

Bloggers for peer-reviewed research reporting, or BPR3, was conceived by scientists and others who write informally about research on the Internet as a simple way to denote that a blog post or website article is discussing peer-reviewed work. Their mission statement: "Bloggers for Peer-Reviewed Research Reporting strives to identify serious academic blog posts about peer-reviewed research by offering an icon and an aggregation site where others can look to find the best academic blogging on the Net."
As well as creating an icon for bloggers to denote when their posts concern peer-reviewed research, the organization will host a central web site where snippets from these posts will be displayed, with links back to the original posts. Readers will be able to choose topics of interest and view only those posts, if they wish.
Eventually, the intention is that bloggers will be able to enter a DOI (digital object identifier) or other unique identifier, and automatically generate code to post the icon, link to the post to the BPR3 site and its aggregation tools, and generate a properly formatted research citation which links to the original article.
Now, the first stage is complete -- the icons have been created (with a lot of helpful input from large numbers of science bloggers). They must be popular, as the site from which people can collect their code has been down for a couple of days due to excessive traffic, presumably. However, it is now up and viewable, so if you want to pick up an icon for your own blog, or just find out more details of the project, please visit this posting. From this post (29 October): "Anyone can use these icons to show when they're making a serious post about peer-reviewed research, rather than just linking to a news article or press release. Within a month, these blog posts will also be aggregated here, so everyone can go to one place to locate the most serious, thoughtful analysis and commentary on the web."
Inevitably, given the engagingly self-referential and gossipy nature of blogging, release of these icons attracted a lot of comment and discussion in the blogosphere. Dave Munger, the driving force behind BPR3, has collected links to all these articles into one post here.
Perhaps in some of this discussion, this question has been addressed already, but as I am afraid I do not have time to read all of the 26 (so far) articles, I raise it here. The icon seems to me an excellent way to indicate that the subject of a blog post is a peer-reviewed research article (so long as there is a mechanism to report abuse and remove the icon from posts that use it incorrectly). And the intentions of BPR3 in providing links to the original research article being discussed are admirable. However, these indicators in themselves do not seem to me necessarily to be an indicator of quality of the blog post itself. On a blog, anyone can write anything about anything, whether or not the topic under discussion is peer-reviewed. A lot of traffic to, or comments on, a blog post is not in itself an indicator of quality. (It could, indeed, be the opposite.)
After the blog posts are themselves linked to the original research article, it will be possible for them to be formally cited, making them eligible for aggregation into citation databases. Blog posts that enter the mainstream scientific debate in this sense would have a quality indicator associated with the post itself rather than the research they describe. Is this the intention?

Posted by Maxine Clarke on November 02, 2007
Categories: Perspectives, Quality and value | Permalink | Comments (4)

The Nature Precedings forum on Nature Network is featuring a stimulating and thoughtful discussion about how to handle pseudoscience postings. Santosh Patnaik writes: "Though Nature Precedings screens submissions for pseudo-scientific content, it is possible for such a submission to get through. An example might be this article on Nature Precedings: this website suggests that the authors are supporting “creationism/intelligent design.” This leads one to wonder how pseudo-science is identified, and what the policy is towards accepted articles that are later identified as pseudo-scientific."
Timo Hannay responds: "Nature Precedings does not accept pseudo-science. Spotting this and other inappropriate content is the job of our curators. In the month since launch they have already filtered out many such examples. In this particular case, the curator handling the contribution was concerned about the nature of the content and so consulted a senior journal editor with very considerable expertise in this area. In their opinion, it is not pseudo-science and is worth posting for community comment."
The discussion that follows touches on how to define and identify pseudoscience; whether preperint server managers should block postings of it; what tools can be used by the community to indicate quality; how an inchoherent writing style can obscure meaning; and the role of scientists in helping to promote "real" science and identify the flaws in pseudoscience, for the wider public.

Posted by Maxine Clarke on October 24, 2007
Categories: Quality and value, Technical solutions | Permalink | Comments (0)

Mark Chillingworth, the Editor of Information Week Review, writes in the October issue that there are debates on how to improve peer review, alludes to a recent PRISM statement about the need to protect it, but that "nowhere is there anyone laying out the true costs of peer review". He suggests that these costs need to be calculated as part of any informed way forward to maximize the benefits of the system.
Martin Blume, then Editor-in-Chief of the American Physical Society, wrote in the first of Nature's web debates on access to the literature in 2001:
"Peer review is expensive, and although reviewing by scientists is voluntary, we need to pay our editorial staff. It is more time consuming and hence more costly to review the 10,000 rejected articles than it is to review those that are accepted. Consideration is being given to other forms of peer review, but no savings are as yet obvious if quality is to be maintained."

Posted by Maxine Clarke on October 05, 2007
Categories: Quality and value | Permalink | Comments (0)

Dr Robert Higgs of the Independent Institute, writes:

Journalists, politicians and advocacy groups refer to “peer-reviewed research” and “scientific consensus” as the authoritative last words on controversial matters involving the natural sciences, from climate change to stem-cell research and genetically engineered foods. But many people have an unrealistic view of how the scientific community actually works.
The peer-review process is not, contrary to popular belief, a nearly flawless system of Olympian scrutiny. Any editor of a peer-reviewed journal who desires to reject or accept a submission can easily do so by choosing appropriate referees.
Unfortunately, personal vendettas, ideological conflicts, professional jealousies, methodological disagreements, sheer self-promotion and irresponsibility are as much part of the scientific world as any other. Peer review cannot ensure that research is correct in its procedures and conclusions. A part of the work in every discipline – from the physical sciences to economics –consists of correcting previous mistakes.
At any given time, “scientific consensus” may exist about various matters. Over time, however, new interpretations, tests or observations may demolish that consensus. For instance, in the mid-1970s, an apparent scientific consensus existed that our planet was about to enter another Ice Age. Drastic proposals, such as exploding hydrogen bombs over polar icecaps to melt them. and damming the Bering Strait to prevent icy waters from entering the Pacific, were put forth by reputable scientists and seriously considered by the US government.
The truth is that scientific research at the upper echelons occurs within a fairly small world. Leading researchers attend the same conferences, belong to the same societies, review one another’s work for funding organizations, and so forth. If you do not belong to this tight fraternity, it becomes extremely difficult to gain a hearing for your work, to publish in a “top” journal, to acquire a government grant, to receive an invitation to participate in a scientific conference, or even to place your grad students in decent positions.
“Scientific consensus” often emerges because the members of this exclusive club, and those who support them, have too much invested in the reigning ideas to let go. In this context, it behooves bright young scientists not to rock the boat by challenging anything fundamental or dear to the hearts of those who constitute review committees of funders or journals. The terms "peer review" and "scientific consensus" often serve to suggest a process of disinterested neutrality and saintly pursuit of truth. Like every other human endeavour, however, science is conducted by people with the full range of human emotions and motives.
Good rules of thumb for the non-scientist might be the following: government-funded research that is used to justify that government’s policy should be suspect, whether or not it’s peer-reviewed; and the research of scientists who appear at press conferences in the company of politicians or activists whose agendas they are there to support should be suspect, whether or not the work upholds the consensus opinion.

Robert Higgs is Senior Fellow in Political Economy at the Independent Institute, editor of the quarterly journal The Independent Review, and the author of Depression War and Cold War, as well as numerous books and more than 100 articles in scholarly journals.

Posted by Maxine Clarke on September 17, 2007
Categories: Quality and value, Systems | Permalink | Comments (5)

Pedro Beltrao of Public Rambling writes a stimulating post exploring whether the editor could be a "value unit" for authors and their publications, independent of the journal. He asks:"could there be freelance editors? Could the editors be separated from the publisher? Imagine [if] I read a paper from a pre-print server, ask some people to peer-review (why would they?) and sell our evaluation to a journal.
Also, can a publisher sell the editorial decision to another publisher? Lets imagine a journal that has a very high rejection rate, the editor asks referees for comments but ultimately the manuscript is rejected. The editor could then ask the authors where they want to send it next and offer to provide the referee report and editorial comments directly to the next journal to expedite the process. Could this journal get paid for this?"
Pedro is not referring here to the transfer system that some publishers, including Nature Publishing Group, offer between their journals, but to the editor taking a role rather like that of an agent in book publishing, as supervisor of the peer-review process and then advocate of the "package" of manuscript and reviews in finding the best possible journal in which to publish the work.

Posted by Maxine Clarke on July 24, 2007
Categories: Quality and value | Permalink | Comments (0)

The pseudonymous FemaleScienceProfessor is also an editor for a journal. She has reviewed its reviewers:

"I did a quick, statistically invalid analysis of the reviewer data for the past year to see whether the time it took a reviewer to complete the review was random or correlated with seniority. My working hypothesis was that younger scientists do quicker reviews. The dataset is sufficiently large to make an analysis like this reasonable, but I wasn't rigorous about tracking down reviewer time-from-Ph.D. data. I put reviewers in one of several bins: postdoc, assistant professor, mid-career, late-career, retired, and I put research scientists into these same bins based on where they would be in terms of time since Ph.D. if they were tenure-track. It's not a perfect system, but I just wanted to get a sense for any trends. The quickest reviewing groups are the early-career and retired scientists. "

More analysis is provided in FemaleScienceProfessor's post, and there are some reactions in the comments to this post. At the Nature journals we do not publish reviewer statistics of this type, nor do we capture information about the reviwers' gender, seniority level and so on. Is there interest from among our peer-reviewers to know these statistics, and to have published the type of information provided by the preliminary results of FSP?

Posted by Maxine Clarke on July 15, 2007
Categories: Quality and value | Permalink | Comments (0)

Among the excellent content on Nature Reports Stem Cells is an especially exciting regular feature, "Inside the Paper", which zeros in on recent scientific papers in the field published in Nature and provides detailed expert comment and author responses drawn from the peer-review process. The editors intend this feature to let readers see the paper’s context, strengths and caveats, as well as make the peer- review process a little more transparent.

Two Inside the Paper features have been published this month. The first, Cloning from Chromosomes, discusses the Article by D. Egli et al., Developmental reprogramming after chromosome transfer into mitotic mouse zygotes, in Nature 447, 679–685 (2007). Read a panel of experts' comments on this interesting article, responses from the authors, and comment yourself on The Niche (the Nature Reports Stem Cells blog).
The second Inside the Paper (21 June), Rewriting in blood, is about a paper by I. M. Samokhvalov et al(Nature 446, 1056–1061; 2007), reporting a surprising origin for blood stem cells. As before, you can read some of the peer reviewer's comments, responses from the authors and invite you to add your contribution on The Niche.

Posted by Maxine Clarke on June 28, 2007
Categories: Quality and value, Systems | Permalink | Comments (0)