Peer-to-peer

In its July Editorial Wouldn't you like to know?, Nature Physics (4, 505; 2008) asks how much of the mass of information available online in encyclopaedic form can be trusted. The Editorial discusses various sources: Wikipedia, of course; Citizendium (with its associated Eduzendium); Scholarpedia ; and a brief mention of Encyclopaedia Britannica, which has just begun experimenting with user-generated input (although not noted in the Editorial).
Scholarpedia is the most recent of these resources, and says of itself that it "feels and looks like Wikipedia - the free encyclopedia that anyone can edit. Indeed, both are powered by the same program - MediaWiki. Both allow visitors to review and modify articles simply by clicking on the edit this article link." Scholarpedia is said to differ from Wikipedia in that each article is written by an expert (invited or elected by the public); anonymously peer reviewed to ensure accurate and reliable information; and has a curator - typically its author -- who is responsible for its content and who has to approve any proposed modifications. The website claims that, by this method, "while the initial authorship and review processes are similar to a print journal so that Scholarpedia articles could be cited, they are not frozen and outdated, but dynamic, subject to an ongoing process of improvement moderated by their curators. This allows Scholarpedia to be up-to-date, yet maintain the highest quality of content."
The Nature Physics verdict? "Expert authorship and curatorship of free online information are indeed welcome. If scientists embrace Scholarpedia, then perhaps the opportunity to make sure that their own favourite area is well represented in its pages — as well as the possibility of citations — will prove sufficient incentive to the hard-pressed experts. The potential is huge, and so is the challenge."

Posted by Maxine Clarke on July 02, 2008
Categories: Technical solutions | Permalink | Comments (0)

Today, 18 June, is the first anniversary of Nature Precedings, where researchers can post their unpublished manuscripts, presentations, posters, white papers, technical papers, supplementary findings and other scientific documents, which can all be "peer-reviewed" online by anyone in the scientific community. (The website was available before June 2007 in 'beta' form.) Santosh Patnaik, a user who periodically tracks Nature Precedings at the Nature Network Nature Precedings forum, estimates that the 500th document will be uploaded some time in the next two weeks.
Because the code for Nature Precedings is freely available, Dr Patnaik has mined some data to chart the growth of the website. His results are presented here, in graphical form. The number of posters and presentations, common when the site first launched, is now barely increasing, whereas the number of manuscript uploads has grown at a steady rate over the past year. The most popular discipline, perhaps unsurprisingly, is bioinformatics, although most other disciplines are also becoming more popular, particularly neurosience, 'evolution and ecology', and chemistry. (For those interested in statistics, Dr Patnaik has also estimated the productivity of Nature Precedings authors.)
One aspect of this type of open peer-review is that discussion is not limited to the English language, even though the language of uploaded documents is in English. The vast majority of comments are, however, in English: here is an example of constructive review in the neuroscience field, Nature Precedings style. There are many other examples: the most active discussions are here, but one can also filter by subject area.

Update: Hilary Spencer and the Nature Precedings team provide a one-year perspective at Nature Network.

Posted by Maxine Clarke on June 18, 2008
Categories: Technical solutions | Permalink | Comments (0)

Stephen T Huang, Maged N Kamel Boulos and Robert P Dellavalle write an article in the June issue of EMBO Reports (9, 496-499; 2008) with the title: Scientific discourse 2.0. Will your next poster session be in Second Life?
From the article:

Certainly, peer-reviewed literature and scientific meetings in the physical world will remain the main modes of distributing scientific information and informal communication. Yet, communication through virtual-world technology might become a useful supplement to the traditional discourse. The particular strengths of this technology include: its potential to share, review and comment on information, both with the public and one's peers; options that allow users to create and develop unique objects, and presentations to educate and inform others and to display data; and, last but not least, the time and cost of bringing people together within and across disciplines can be reduced.
As with any new technology, there are issues that could have an impact on the usefulness of online communication and its acceptance within the scientific community. Scientists who rely on peer-reviewed data for their work might find Web 2.0's lack of proofreading unacceptable to document research findings. However, we should explore the existing and potential applications of virtual communication for unique ways to discuss ideas, answer questions, educate and debate. Our ability to understand what we can accomplish in online worlds depends on our collective experience with the technology. The more scientists and clinicians who work with and comprehend the applications of virtual worlds for their respective research fields, the sooner we will realize how this technology can be best applied. The next step is to invite ourselves into these online realms, experiment with what they have to offer, and see where our exploration and creativity takes us.

(Stephen T. Huang is at Northwestern University's Feinberg School of Medicine in Chicago, Illinois, USA; Maged N. Kamel Boulos is at the University of Plymouth, UK; Robert P. Dellavalle is at the Denver Veteran's Affairs Medical Center and the University of Colorado Denver, Denver, Colorado, USA.)

Posted by Maxine Clarke on June 10, 2008
Categories: Technical solutions | Permalink | Comments (0)

Noah Gray, an editor at Nature Neuroscience, asks at Action Potential blog why neuroscientists are passing on the seemingly golden opportunity to communicate with one another online, for example on published articles at a journal website, or in an online journal club. Many have expressed opinions about reasons for this reticence: Noah links to some articles in his well-argued post, and you can read other thoughts (and find links to some of the same articles) at Nature Network (for example at Gobbledygook and at Flags and Lollipops).

Here, I'd like to highlight a response by "Michael" in the comment thread to Noah's Action Potential post, as I believe this summary covers many, if not all, of the reasons why scientists in a discipline tend not to post comments on, and discuss, scientific research papers online. "Michael" writes:

I think there are a number of reasons as to why “Web 2.0” has not played much of a role in discussing neuroscience research. In no particular order:
1) Inefficiency: If I want to know why somebody used buffer x for their biochemistry experiments, or why they didn’t do control experiment xyz I email the authors, or use the phone if I know them. Why post it in the comment section of their paper, and wait for 5 weeks until they bother to check? And why does everybody else need to know about it?
2) Lack of dialogue: Commenting forums are poor venues for true dialogue. If you analyse the comment sections for the more popular entries, either here or elsewhere (for example the New York Times) there rarely is a true back-and-forth of ideas. All too often it’s 50 comments trying to be as witty as possible, with few people attempting to follow and respond to what others have written. There are rare cases where a small number of like-minded people post thoughtful comments at just the right rate to allow for a meaningful discussion. Without some type of moderation, it will remain the exception. The dynamics of a true group discussion, be it at a Journal Club or at your poster, are often taken for granted, but can’t be easily replicated online.
3) Who is commenting: It’s fine to have a democracy of opinions: in science, I don’t care so much about it. The people I want to hear/read often choose not to comment, whereas others who have nothing to say keep posting.
4) The Fear Factor: This one is obvious, and is why scientists have lab meetings or face-to-face Journal Clubs. It’s also why attempts to discuss papers online haven’t quite lived up to expectation. Ideally, we want to be honest in our opinion of a paper, but we are also human and don’t want to suffer the consequences of bruising the ego of a potential reviewer or search committee member. Staying anonymous is not the solution, since that makes it difficult for everybody else to properly evaluate the comment. After all, it does matter who is doing the criticizing.
5) Speed: Even the liveliest online discussion of a paper will drag on over hours or days. If a paper grabs my attention I will discuss at a lab meeting or Journal Club and over the course of one hour we will have thoroughly dissected it. Our attention span only lasts so long. Again, the online dynamics of who logs in at what time don’t allow for a true discourse that leads to some sort of resolution.

Posted by Maxine Clarke on March 17, 2008
Categories: Technical solutions | Permalink | Comments (1)

The Nature Precedings forum on Nature Network is featuring a stimulating and thoughtful discussion about how to handle pseudoscience postings. Santosh Patnaik writes: "Though Nature Precedings screens submissions for pseudo-scientific content, it is possible for such a submission to get through. An example might be this article on Nature Precedings: this website suggests that the authors are supporting “creationism/intelligent design.” This leads one to wonder how pseudo-science is identified, and what the policy is towards accepted articles that are later identified as pseudo-scientific."
Timo Hannay responds: "Nature Precedings does not accept pseudo-science. Spotting this and other inappropriate content is the job of our curators. In the month since launch they have already filtered out many such examples. In this particular case, the curator handling the contribution was concerned about the nature of the content and so consulted a senior journal editor with very considerable expertise in this area. In their opinion, it is not pseudo-science and is worth posting for community comment."
The discussion that follows touches on how to define and identify pseudoscience; whether preperint server managers should block postings of it; what tools can be used by the community to indicate quality; how an inchoherent writing style can obscure meaning; and the role of scientists in helping to promote "real" science and identify the flaws in pseudoscience, for the wider public.

Posted by Maxine Clarke on October 24, 2007
Categories: Quality and value, Technical solutions | Permalink | Comments (0)

Nature Precedings is now out of "beta testing" and is launched. This new community service is described at Nautilus, the NPG blog for present and future authors. Submissions are screened by our professional curation team for relevance and quality, but are not subjected to peer review. High-quality contributions from biology, medicine (except clinical trials), chemistry and the Earth sciences are welcomed.

More details about Nature Precedings can be found here. Because they have not been peer-reviewed, many of the findings you read at Nature Precedings may be preliminary or speculative, and remain to be confirmed. Please bear this in mind when deciding how seriously to take them.

Submissions are not accepted from fields in the physical sciences that are are already well served by preprint servers such as arXiv.org. Content that considered to be non-scientific or pseudoscientific is rejected. We accept only genuine contributions from qualified scientists. This will usually require submitters to have a recognized academic affiliation. Incomplete submissions will also be rejected. This is a free service, so please help us to help you by completing all relevant sections of the submission form.

The reactions of scientists and publications to the launch of Nature Precedings can be found at this Connotea page, which is regularly updated. Here are one or two such articles:
Nature Precedings pre-print server for biomedical research
Nature Precedings: A nicer version of ArXiv[e] for biomedical research
Chemistry Central: A new preprint server from Nature.

Posted by Maxine Clarke on June 18, 2007
Categories: Open peer review, Systems, Technical solutions | Permalink | Comments (0)

See Nature Reports Stem Cells for a fresh perspective on a paper reporting a technique for cloning from zygotes. In this fortnight’s Inside the Paper, a new form of scientific reporting pioneered by Nature Reports Stem Cells, you can read a moderated discussion between the authors and the paper's peer-reviewers. Readers can learn what the foremost experts in the field had to say about the submitted paper’s strengths and shortcomings. See what the authors saw, and read their responses as they revised their paper for eventual publication in Nature. You can also add your own comments on The Niche, the Nature Reports Stem Cells blog.
Featured paper: D. Egli et al. Developmental reprogramming after chromosome transfer into mitotic mouse zygotes. Nature 447, 679–685 (2007).

Posted by Maxine Clarke on June 13, 2007
Categories: Systems, Technical solutions | Permalink | Comments (0)

FCW.com News - Data explosion strains peer review

The link above is a stimulating little article on FCW.com about how the US National Institute of Standards and Technology (NIST) deals with the overload of thermodynamic data reported in journals. From the FCW article:

"Recent improvements in measurement equipment mean that an already voluminous amount of thermodynamic data is doubling every 10 years.
That explosive growth is straining the traditional journal-based peer-review system and causing increasing numbers of errors to creep into the data. Companies in the chemical, pharmaceutical and energy industries depend on accurate data for their engineering applications and research projects.
As part of its responsibility for promoting U.S. competitiveness through standards and technology development, NIST worked with industry partners to create a standard data format and online system for verifying and disseminating thermodynamic data."

The XML-based system transforms the published data into a standard format and stores it in a central database that researchers can access via the Web. When authors submit data, the system automatically checks for inconsistencies and alerts the authors to any questionable data.

Posted by Maxine Clarke on June 04, 2007
Categories: Technical solutions | Permalink | Comments (0)

south by southwest festivals conferences
At the link above is a podcast of a session from the SXSW (South by South-West) Interactive Festival in Austin, Texas, held in March. From the conference website: "New publishing technologies challenge the traditional structure of peer-reviewed scientific journals. For hundreds of years the "article" has been the primary vehicle for conveying scientific information - but semantic markup, tagging and wiki are reconstructing scientific publications into a flexible and evolving concept." The panel looked at the social and legal implications of "Web 2.0" and the "Semantic Web" as they impact science and scientific knowledge. The moderator was John Wilbanks, Executive Director of Science Commons, Creative Commons, and one of the invited speakers was Timo Hannay, Director of Web Publishing of the Nature Publishing Group. The podcast has just been uploaded to the SXSW site, and is freely available by going to the link at the top of this post.

(Cross-posted on Nautilus.)

Posted by Maxine Clarke on June 04, 2007
Categories: Technical solutions | Permalink | Comments (0)

Nature’s Journal Club

Our latest blog (link above) is the blog for the Journal Club, a weekly column published in Nature’s Research Highlights pages. Each column presents a researcher’s choice of reecnt paper, explaining the reason why he or she is enthused about it. At the Journal Club blog we invite readers to discuss the subjects raised in the columns. Please do take a look at the entries on the blog, and, as my colleague Oliver Morton puts it, "enrich their comment threads with your insight and speculation."

(Cross-posted on Nautlius, the author blog.)

Posted by Maxine Clarke on May 29, 2007
Categories: Technical solutions | Permalink | Comments (0)

C. Cristofre Martin and Kenneth B. Storey write:
Scientists are often involved in the peer review of grant applications and/or manuscripts submitted for publication. We rely on the anonymity of the system to allow us to be completely frank and unbiased in the comments that we provide to the author(s) of the article or grant.
However, we wish to point out an alarming situation. The now heavy reliance on electronic means of communicating between reviewers and publishers/granting agencies in the form of MS Word and other electronic documents, generated PDFs, and other user-generated file formats has created in a situation where anonymity can no longer be ensured.
The reason for this breach in security is that most state-of-the-art software applications will embed information about the creator of the document with the normally invisible metadata of the file. This metadata can be viewed by means as simple as opening the file within a text editor application or by viewing the creator information for a file within the operating system such as 'Get Info' in the Mac world or 'Properties' on PCs. Typically, the source of this metadata is the user account information that is associated with the specific computer being used to generate the document.
Authors, journal editors, publishers and granting agencies need to be cautious about how 'anonymous' information is transmitted between the creator and the recipient. Creators of anonymous documents should check that the programs that are used to create their documents are secure and if not, adjust security settings where possible or delete creator information in the file properties before sending off their reviews. Publishers and granting agencies should also consider adjusting dissemination methods such that original reviewer-created documents are never forwarded directly to authors. The greater use of Web forms for both the input and transmission of reviews is one obvious solution.

C. Cristofre Martin, Department of Biochemistry, St. George's University, St. George's, Grenada , West Indies.
Kenneth B. Storey, Department of Biology, Carleton University , Ottawa , Canada.

[Note from Maxine: For information, Nature Publishing Group journals use a Web-based peer-review system to ensure anonymity, as do many, but not all, other publishers. The Nature journals also require large datasets and other supplementary information to be deposited by sumbitting authors into a public database or supplied on CD/DVD for the purposes of peer-review. We do not allow authors to post such supplementary information solely on their own or their institutional websites, partly for the reasons outlined by Drs Martin and Storey. Further details of our policies can be found at the NPG authors and reviewers' website]

Posted by Maxine Clarke on May 22, 2007
Categories: Technical solutions | Permalink | Comments (0)

In A Blog Around The Clock : Two Cultures, Coturnix writes about his surprise that the humanities seem more reluctant to experiment with peer-review systems than the scientific community. Coturnix describes his experiences of selecting posts for blog "carnivals" (themed collections of posts on one blog by various authors), and suggests that a peer-reviewed online blog-journal is the next logical step, or as he puts it: "I'd love to see publication of blogging anthologies collecting the best annual output by medical, environmental, education and humanities bloggers."

Paolo Massa has collected some links to articles with a similar theme.

Posted by Maxine Clarke on April 16, 2007
Categories: Systems, Technical solutions | Permalink | Comments (0)

The Species 2000 & ITIS Catalogue of Life will become the comprehensive catalogue of all known species of organisms on Earth. Started in 2000, it is hoped to be complete by the year 2011. As things stand, the catalogue has just reached 1,008,965 species: probably just more than half of the world's known species. (The final total is expected to be around 1.75 million.)

The catalogue is compiled with sectors provided by 47 taxonomic databases, many containing data and opinions from extensive networks of specialists, so that the complete work contains contributions from more than 3,000 specialists from throughout the taxonomic profession. These databases are peer-reviewed by teams from the Species 2000 and ITIS programme, who also select appropriate sectors and integrate them into a single coherent catalogue with a single hierarchical classification.

It is planned to introduce alternative taxonomic treatments and alternative classifications, but an important feature is that for those users who wish to use it, a single preferred catalogue, based on peer reviews, will continue to be provided.

Posted by Maxine Clarke on April 12, 2007
Categories: Technical solutions | Permalink | Comments (0)

Richard Akerman

Peer review needs to adapt to the pace and volume of information published online

How does the role of peer review evolve when the body of scholarly knowledge expands from slowly circulating, static documents to the universe of rushing, dynamic interactions made possible by the Internet? Although traditional forms of scholarly communication are still used, the sheer volume and pace of information enabled by the Internet and publishing tools such as weblogs (blogs) demands novel solutions.

Continue reading "Technical solutions: Evolving peer review for the internet" »

Posted by Maxine Clarke on June 20, 2006
Categories: Peer review debate, Technical solutions | Permalink | Comments (0)

Chris Anderson

Scientific publishers should let their online readers become reviewers.

Who are the peers in peer review? In journals such as Nature, they usually have a PhD and work in a field relevant to the paper under consideration. If they are academics, they may be tenured professors, usually people on a relatively short list of experts who have agreed to review papers. This is a little élitist, but credentials such as PhDs and tenure are given in part to reward those things – experience, insight, brains and the respect of other researchers – that also make for wise advice. The process is not perfect, for reasons ranging from cronyism to capriciousness, yet long experience has shown it to be better than the alternatives.

Continue reading "Technical solutions: Wisdom of the crowds" »

Posted by Maxine Clarke on June 14, 2006
Categories: Peer review debate, Technical solutions | Permalink | Comments (9)

Herbert Van de Sompel

What functions do we take for granted in print?

The Digital Library Research and Prototyping Team at the research library of the Los Alamos National Laboratory conducts research on various aspects of scholarly communication in the digital age, including peer review. Our research attempts simultaneously to analyse properties of the existing review system, and to formulate feasible alternatives.

Continue reading "Technical solutions: Certification in a digital era" »

Posted by Maxine Clarke on June 05, 2006
Categories: Peer review debate, Technical solutions | Permalink | Comments (1)