Peer-to-Peer

Systems: Online frontiers of the peer-reviewed literature

Theodora Bloom

The Internet is allowing much more interactive science publishing

Online tools can be used to improve the accuracy, transparency and usefulness of the scientific literature by moving away from the traditional emphasis on closed peer review. Given the capability for post-publication amendment of articles, the scientific articles themselves and the peer-review process will soon be profoundly different from today’s standard.


The traditional system of peer review entails careful scrutiny of each submitted article by at least two experts in the field, with the reviewers’ names withheld from the authors and an editor deciding, in the light of reviewers’ comments, whether to offer publication. The Internet is encouraging authors, editors and publishers to experiment with publishing models that deviate from this system.

Innovation in peer review

In addition to assessing scientific accuracy, many reviewers are also asked to judge whether a manuscript reaches the level of interest appropriate for a given journal. Some journals — such as the 60 titles in the BMC series published by BioMed Central — pledge to publish all sound research within the journal’s scope, and not to judge by interest level. Does asking reviewers to judge primarily whether the work is sound, rather than filtering from among sound papers, affect how the journal is seen in the field and its success?

Almost all the BMC journals were launched at the same time, share open-access goals, and aim to cover similarly sized portions of the scientific literature, as well as sharing almost identical systems of peer review and having similar acceptance rates among submitted articles, but their ‘success’ — measured by submission, publication and citation rates — shows significant variability. For example, BMC Bioinformatics and BMC Public Health each receive hundreds of manuscripts each month and publish more than a hundred, whereas BMC Evolutionary Biology and BMC Family Practice receive and publish only tens of articles each per month. Given that the peer-review processes of these journals are similar, what is the reason for the variation?

One key difference in the peer review operated by two categories of BMC journal — ‘medical’ and ‘biological’ — is that the medical journals operate ‘open’ peer review, where reviewers are known to the authors and, in the event of publication, the reviewers’ names and reports are published with the article, whereas the biological journals operate more traditional ‘closed’ peer review, in which reviewers’ names are withheld from authors. Our experience with the BMC series bears out the BMJ’s previous experience that peer reviewers are more reluctant to serve in an ‘open’ system. (See article in this debate by Groves. We have found, however, that there can be greater variation in the proportion of invited reviewers who agree to serve between two BMC-series journals that operate open peer review, or two journals that operate closed peer review, than between all open-review and all closed-review BMC-series journals.

A recent study of BMC journals (E. Wager et al. BMC Medicine 4, 13;2006) shows that author-nominated peer reviewers provide reports of similar quality and tone to reports by editor-chosen reviewers (although the former are less likely to recommend rejection during the initial round of review) . In this study, ‘quality’ was assessed blind by two raters, using seven indicators of review quality (such as ‘assessing originality’ or ‘providing constructive comments’), each rated on a five-point scale.

A further experiment in open peer review, with author-nominated reviewers and entirely open communication between the author and reviewers (with very little intervention by editors), is being undertaken by the journal Biology Direct (see article in this web debate by Koonin et al. )

Use of the Internet also means that peer review need no longer be a one-step process: preprint servers and online comment and reviewing systems allow an article to accrue commentary and analysis after it is published. This ranges from the ‘post a comment’ option operated by many online journals to the two-stage system used by Atmospheric Chemistry and Physics (see article in this web debate by Koop and Poschel. Further and wider experimentation in this direction has been urged by Van de Sompel and Ackerman elsewhere in this debate.

Definitive version?

But if an article can be amended and commented on after publication, what should be seen as the definitive version? Some journals, like FASEB Journal and Genome Biology, already provide different versions of a single article for print and online audiences. Indeed, any article that includes a database or movie available only online can be considered in this light. Many traditionalists object to the idea of multiple versions of scientific articles, condemning them as duplicate publication of the same data. But users of software are familiar with the idea that version 1 is modified through versions 1.1, 1.2 and so on, until a new version 2 is released. If some data have been wholly reanalysed, or indeed if a research article describes a software tool rather than a data set, is it not reasonable to publish a second article describing progress at some point, or a new version of the first article? For example, Nucleic Acids Research publishes an annual ‘Databases’ issue that this year included articles describing updates to 68 databases. These articles are seen as wholly new, but each could in fact be described as an update of the previous year’s description. Indeed, it might be easier to navigate the literature were such articles to be more explicitly branded as such.

In considering how such new versions and updates should be peer reviewed, most journals would expect to operate stringent peer review for a completely new article describing a whole new version of a piece of software or new analysis of data, but many would use a slightly more relaxed consultation with expert reviewers when an addendum or correction is made, by using advice from only a single reviewer, and/or looking only for exceptional reasons not to publish (see article in this debate by Riley).

The traditional one-version-only view of a research article as a snapshot of the information about a particular problem is also challenged by large-scale studies, such as clinical trials in medicine or genome-wide analyses in biology, descriptions of which almost always constitute a ‘work in progress’ rather than a definitive and final view.

Theodora Bloom has editorial responsibility for biological publishing at BioMed Central. She has a PhD in developmental biology and post-doctoral experience in cell biology. She joined the editorial staff at Nature, then moved to Current Biology, and became publishing manager for Current Biology and its siblings. From Elsevier Science London she rejoined the Current Science Group in 1999, shortly before it launched BioMed Central.

Read more See this article in Nature’s web focus here

Comments

  1. Report this comment

    Pietro Ghezzi said:

    Concerning Dr. Bloom’s article on the use of online systems, I would maintain that the internet has actually slowed down many procedures, for us authors and reviewers. Although it made it certainly easier for publishers.

    The current methods of online submission, used now by most journals, ask the authors to do all the work. Spending much more time that when we used to submit 3 printed copies. We have to go through a series of pages and not just upload our files. The most annoying for me is to retype all author names, often having to wait for the page to reload after we add each author. This clearly saves money to the publisher, but the procedure is used also by journals that either make money or ask us to pay page charges.

    Now, this online system is extended to reviewers. When a journal ask me to be a refereee for a paper, I receive an email with URL, username and password. To proceed I often have to go through a process of updating the username and password to download the manuscript file. This often requires filling a form on my position, interests, etc. that is clearly intended for marketing purposes, like those you fill to get a catalog from a commercial company.

    I think this is too much. In the end I have to spend much more time to do a review. Needless to sayobody wants to go back to the old ways of sending by courier printouts. E-mail is fine.

    I normally respond to the invitation to review a manuscript online by asking to email me the pdf of the manuscript and a word file of the reviewer form.

    I strongly encourage those that share my irritation to do the same. We can go though a sort of suffering to publish a paper, but to do so when we help the journal by acting as a referee is masochistic.

  2. Report this comment

    Georgii Alexandrov said:

    I would like to emphasize the words of Dr Bloom that show some inadequacy of the Nature experiment.

    “In addition to assessing scientific accuracy, many reviewers are also asked to judge whether a manuscript reaches the level of interest appropriate for a given journal. Some journals — such as the 60 titles in the BMC series published by BioMed Central — pledge to publish all sound research within the journal’s scope, and not to judge by interest level. Does asking reviewers to judge primarily whether the work is sound, rather than filtering from among sound papers, affect how the journal is seen in the field and its success?”

    The most vulnerable part of Nature’s peer review system is evaluation of the “level of interest”: a manuscript submitted to Nature is sent to peer-review only after it was found to be interesting enough. Open peer-review, or it might be beter to name it as “peer-voting” (http://www.cbmjournal.com/content/1/1/10), may be really needed to substantiate editorial decisions of this sort.

Comments are closed.