Peer-to-peer

From Nature 454, 928; 21 August 2008:
The US Federal Bureau of Investigation (FBI) plans to publish in peer-reviewed journals much of the scientific evidence it used to pin the 2001 anthrax attacks on microbiologist Bruce Ivins.
Ivins's suicide on 29 July means that the government's case against him will never be heard in court. The trickle of circumstantial evidence released in an investigation that had previously fingered the wrong man has lawmakers, scientists and others clamouring for more information.
In response, the FBI invited scientists and journal editors to a briefing in Washington DC on 18 August to discuss the science of the case and investigators' conclusion that a single man carried out the multiple, deadly mailings of anthrax spores. But FBI officials admit that some mysteries of the case may never be resolved. "I don't think we're ever going to put the suspicions to bed," said Vahid Majidi, assistant director of the division of weapons of mass destruction at the FBI. "There's always going to be a spore on a grassy knoll."
In lieu of expert witnesses and cross-examinations, the FBI plans to offer the evidence for peer review and will keep much of the data quiet until they are published. FBI laboratory director Chris Hassell anticipates a dozen or so papers related to the case, in addition to those that have already been published. However, Hassell says, some details of the investigation will remain confidential, so that potential bioterrorists won't know exactly what they're up against. "It's just what we have to do for national security," he says.
"Given that Ivins cannot stand trial, putting the data through the rigorous process of scientific review may be the best available alternative," says Alan Pearson, director of the biological and chemical weapons control programme at the Center for Arms Control and Non-Proliferation in Washington DC.

Continue reading "Science of anthrax case will be submitted to peer-review" »

Posted by Maxine Clarke on August 21, 2008
Categories: Ethics, Journals, Policy and Public | Permalink | Comments (0)

From Nature News in Brief (454, 564; 2008):
The system used by the US National Institutes of Health (NIH) to evaluate grant proposals does not adequately compensate for reviewer bias, affecting one in four proposals, a study finds.
Valen Johnson, a biostatistician at the University of Texas M.D. Anderson Cancer Center in Houston, evaluated reviews for nearly 19,000 grant proposals performed by around 14,000 reviewers in 2005 (V. E. Johnson Proc. Natl Acad. Sci. USA doi:10.1073/pnas.0804538105; 2008).
Each application is typically read by 2–5 reviewers, then discussed when a larger study section of about 30 reviewers meets. In the end, scores from all the study section's members — readers and nonreaders alike — are averaged together.
The system fails to account for individual bias and places undue weight on panel members who have not read the proposals, Johnson argues. He found that the top grants were largely unaffected by reader bias, but that it did affect grants closer to the funding cut-off line. Overall, accounting for reader bias changed about 25% of the funding decisions, meaning that one in four funded proposals would have been replaced by one that had not been funded.

See here for a longer version of this story, which includes an informative online readers' discussion.

Posted by Maxine Clarke on August 14, 2008
Categories: Ethics, Grants, tenure and positions | Permalink | Comments (0)

From the Editorial in the August issue of Nature Reviews Cell and Molecular Biology (9, 583; 2008):

First-time applicants [for NIH grants] compete with thousands of new and established scientists, an experience that can be intimidating and frustrating. The level of detail required for the majority of applications — for example, the exhaustive budgetary specifications — has been a burden not only for applicants, but also for reviewers. Moreover, a grant proposal can take up to 18 months to pass through the system, waiting in line behind older applications, most of which must go back and forth to applicants for amendments before approval. Of course, funding is not guaranteed: the NIH receives between 35,000 and 40,000 proposals a year, of which only 25–30% will eventually be funded.
On the basis of feedback solicited from the life-sciences community on the current peer-review system, the NIH plans to revamp the grant-review process to encourage innovative research and reward quality science. The new practice involves shortened applications that should improve the value and transparency of the review process and ensure balanced, fair reviewing across scientific fields and scientific career stages. Recruiting, training and rewarding the best reviewers is also a priority. The new process allows reviewers to focus more on the science presented and less on the details of financial requests, and compensates them for their time and effort.
Hopefully, the NIH's changes will not only foster new innovative research efforts, but they will also free up some of scientists' precious time — allowing them to spend more time planning experiments, on the bench and writing and reading scientific papers.
See also: Will NIH's overhaul be cosmetic or curative?

Posted by Maxine Clarke on August 06, 2008
Categories: Grants, tenure and positions | Permalink | Comments (0)

In a discussion at Nature Network about the desirability, or otherwise, of developing a set of metrics to measure individual value, Roberto Cerbino suggests that an interesting factor for an experiment is peer-reviewing activity. Some journals already publish at the end of the year the list of names of reviewers. Perhaps, he writes, they could add some quantitative factor such as the number of papers reviewed or an evaluation index of the reviewer’s work? This would be a small but useful step to assess the contributions of individuals to their fields of activity.
Raf Aerts disagrees, cautions that researchers could accept as many review requests as possible to increase their r-factor, including manuscripts that are not really in their field of expertise.
Thomas Lemberger, EMBO editor of Molecular Systems Biology, on the other hand, points out that a simple ranking of referees could be based on combining frequency of review with speed: referees who review frequently and fast (which he calls the FF-metric) for a given journal are likely to be good referees, at least within the field covered by the journal.

Posted by Maxine Clarke on July 30, 2008
Categories: Journals | Permalink | Comments (0)

Charles Darwin comments on the latest UK government initiative to engage society as a whole with science: "Scientists pressed, sweating into corners as costermongers, corn-chandlers, dogs meat men, chimneysweep’s boys, executioner’s assistants, crimps, pimps, organ grinders, grooms of the stool, fullers, gentlemen of the road, members of the aristocracy and ladies of the night (to mention but a few) all clamour to press on you their views on string theory, stem cell therapy, plate tectonics or catalytic cracking.
The government has called for Society to have its say on science. I hope your supervisors and lab managers will be patient as you listen to the throngs at the gates of your institutions, and that you will wear extravagent disguises when in public to avoid molestation by the public anxious to press their scientific opinions on you."

On a more serious but less entertaining note, the Science and Society Strategy initiative aims to capture a range of views from the general public, scientists, businesses, media, education and government. It seeks input on three key areas:
* How to improve communication, generate interest, increase participation and convey the relevance of science;
* How to build trust and confidence in scientific research in the public and private sectors; and
* How to inspire young people from diverse backgrounds to become tomorrow's skilled scientists.
In what is believed to be the first website of its type for a Government consultation, an online interactive hub has been established to capture the feedback. It features discussion forums, videos and an interactive consultation document allowing visitors to respond to the entire strategy or to the perspectives of invited scientists, science communicators, broadcasters, policymakers and businesspeople on topics such as science teaching, how to reward good "public engagement" work and how to develop scientific literacy. The consultation will run until 17 October 2008. It is not clear from the website how the responses will be assessed. As well as commenting at the Science and Society Strategy website, you are also very welcome to contribute to the online conversation at Charles Darwin's blog.

Posted by Maxine Clarke on July 21, 2008
Categories: Policy and Public | Permalink | Comments (1)

On the topic of the ‘data deluge’, Sarah Kemmitt notes at Nature Network that the UK Government has opted for an increasingly used technique (see, for example, Elsevier’s Grand Challenge) to scope ideas for a strategy for how to make best use of interrelated information.
Sarah refers to the British Cabinet Office’s Power of Information Taskforce project ‘Show Us a Better Way’, which is asking for suggestions to develop better ways to publish the vast swathes of non-personal information that the government collects and creates, using the incentive of a competition (here is a BBC article about the initiative). From the Show Us a Better Way website:

Ever been frustrated that you can't find out something that ought to be easy to find? Ever been baffled by league tables or 'performance indicators'? Do you think that better use of public information could improve health, education, justice or society at large?
The UK Government wants to hear your ideas for new products that could improve the way public information is communicated. The Power of Information Taskforce is running a competition on the Government's behalf, and we have a £20,000 prize fund to develop the best ideas to the next level. You can see the type of thing we are are looking for here.
To show they are serious, the Government is making available gigabytes of new or previously invisible public information especially for people to use in this competition.

Sarah is part of the British Library TalkScience team, and is a co-founder of the Nature Network group Scientific Researchers and Web 2.0: Social Not-working? All are welcome to join the group and contribute to the conversation, in advance of a meeting in September for a focused discussion of the topic.

Posted by Maxine Clarke on July 09, 2008
Categories: Technical solutions | Permalink | Comments (0)

In its July Editorial Wouldn't you like to know?, Nature Physics (4, 505; 2008) asks how much of the mass of information available online in encyclopaedic form can be trusted. The Editorial discusses various sources: Wikipedia, of course; Citizendium (with its associated Eduzendium); Scholarpedia ; and a brief mention of Encyclopaedia Britannica, which has just begun experimenting with user-generated input (although not noted in the Editorial).
Scholarpedia is the most recent of these resources, and says of itself that it "feels and looks like Wikipedia - the free encyclopedia that anyone can edit. Indeed, both are powered by the same program - MediaWiki. Both allow visitors to review and modify articles simply by clicking on the edit this article link." Scholarpedia is said to differ from Wikipedia in that each article is written by an expert (invited or elected by the public); anonymously peer reviewed to ensure accurate and reliable information; and has a curator - typically its author -- who is responsible for its content and who has to approve any proposed modifications. The website claims that, by this method, "while the initial authorship and review processes are similar to a print journal so that Scholarpedia articles could be cited, they are not frozen and outdated, but dynamic, subject to an ongoing process of improvement moderated by their curators. This allows Scholarpedia to be up-to-date, yet maintain the highest quality of content."
The Nature Physics verdict? "Expert authorship and curatorship of free online information are indeed welcome. If scientists embrace Scholarpedia, then perhaps the opportunity to make sure that their own favourite area is well represented in its pages — as well as the possibility of citations — will prove sufficient incentive to the hard-pressed experts. The potential is huge, and so is the challenge."

Posted by Maxine Clarke on July 02, 2008
Categories: Technical solutions | Permalink | Comments (0)