Soapbox Science

A Review of Peer Review

Josh Salvi is a biomedical fellow in the Laboratory of Sensory Neuroscience at The Rockefeller University and a student in the Tri-Institutional MD-PhD Program. He also acts as Executive Director of the Weill Cornell Community Clinic. You can read more of his posts on his blog, Musings of a MudPhud and can follow him on Twitter (@joshsalvi).

A key component in science is communication. We hope that this communication is accurate, conveys its intended purpose, and remains archived for future reference. Thus, the medium by which this message is conveyed must be regulated.

Peer review is the process by which members of a field evaluate the work of other members in the same field as a form of regulation. This increases credibility and, presumably, quality within the field. For example, this can refer to review of manuscripts for publication, review of teaching methods by other educators, or, within the medical profession, the creation and maintenance of health care standards. My focus will be on scholarly peer review, more particularly on methods of peer review in publication and less in the clinical setting for the purposes of this post. Issues relating to technical peer review in fields such as engineering or standardization within education will not be discussed here. However, remember that “peer review” is a broad term encompassing many fields. The purpose of this post is to bring to light historical context and to bring into focus the benefits and drawbacks of our current system.

In 1665, Henry Oldenburg created the first scientific journal to undergo peer review, the Philosophical Transactions of the Royal Society. Peer review in this journal differed from the kind we see today. Whereas professionals in the same field and often in competing labs will review today’s articles for publication, articles in this journal were reviewed by the Council of the Society. This journal created a foundation for the papers we see today, disseminating peer-reviewed work and archiving it for later reference. Peer review later developed in the 18th century as one where other professionals, often experts in the field, would perform the review, as opposed to the editorial review of the aforementioned journal. This form of scholarly peer review did not become institutionalized until closer to the 20th century. However, professional peer review, such as that performed by physicians, dated back to the 9th and 10th centuries, where one physician would comment on the ethical decisions or procedures of another.

Since that time, scholarly peer review has become a mainstay of academic publication. It is amazing to think that this regulatory process has only been so strong for less than a century. However, the procedure does not come without significant criticism (Though what topic in science is not heavily criticized?).

First, though, let us consider the benefits of scholarly peer review. Mentioned above was the improved quality of published work. Simply put, this works by first presenting a barrier that authors must overcome in order to be published, and critiques from reviewers are then addressed by authors to improve the quality of a manuscript. These suggestions may include additional experiments that will further test the work. The process filters out scientific error, thus improving accuracy of published information. Poor-quality work is rejected by the peer-review process. Additionally, work is stratified by journal quality, and this process routes papers to the correct tier. In effect, peer review is at the heart of scientific critique.

One of the most common critiques of peer review is that it remains untested, as purported by a 2002 article in JAMA. The Cochrane Collaboration in 2003 (and reconfirmed in 2008) concluded that there existed “little empirical evidence to support the use of editorial peer review as a mechanism to ensure quality of biomedical research, despite its widespread use and costs.” They recommend, “A large, well-funded programme of research on the effects of editorial peer review should be urgently launched.” Additionally, a study took an article about to be published in the British Medical Journal (BMJ), purposely added a number of errors, and measured the error detection rate to be about 25%, with no reviewer correcting more than 65% of the errors. This study was particularly interesting, as it was headed by Dr. Fiona Godlee, who later went on to critique the lack of external peer review of the Cochrane Collaboration. Her pioneering work in this field has stimulated much interest.

Finally, single-blinded peer review is open to bias. This could be bias against nationality, language, specialty, gender, or competition. Additionally, there is a common trend of bias toward positive results. Double-blinded review may help to overcome this critique.

Alternatives to single-blind review include double-blind review, post-publication review, and open review. In double-blind review, neither the authors nor the reviewers know the other party, and this would presumably reduce aforementioned bias. Surveys had shown a preference to double-blind review. Post-publication review would be an excellent supplement to the current review system to improve the rate of error correction in publications. Finally, open peer review, where the reviewer is known, would also possibly reduce the bias. However, one may be less willing to critique work by a senior author in the field, and the pilot by Nature in 2006 was far from successful.

The question is not, “Is peer review an ineffective system?” I believe it is. Instead, the question is, “Why does peer review sometimes fail to meet our lofty expectations?” This is a question that can be answered with rigor.

At this stage, the system is the best we have, and problems lie less in the peer review process and more in the access to scholarly work without a costly subscription. Discontent in the field does not translate to a desire for one of the alternative methods described. Nonetheless, we should be critical of our process, much in the same way the process itself is critical.

Comments

  1. Report this comment

    Phil Davis said:

    I’m a little confused with your final assertion that peer review (whether pre- or post-publication) may experience an access bottleneck. A randomized controlled trial of open access publishing in 36 scientific, medical, social science and humanities journals finds no citation difference when articles are made freely available. If citations are an indication of access and incorporation from scientific peers, then a large citation advantage to free articles would have been apparent within the observation period (3-years). We did find that freely-accessible articles were downloaded more often from a larger population of potential readers. These findings imply that research scientists (the community of peer-reviewers) do not experience the dearth of access you claim.

    Increasing access to the scientific literature may help many readers disenfranchised by the traditional access model, yet it may do nothing to improve the system of peer review.

    [1] Davis PM. 2011. Open access, readership, citations: a randomized controlled trial of scientific journal publishing. The FASEB Journal 25: 2129-34. http://dx.doi.org/10.1096/fj.11-183988

  2. Report this comment

    Joshua Salvi said:

    This is an excellent point. I must clarify myself with that statement. I hope my response does not distract from the main point of the article: the current peer-review system and its alternatives.

    As you point out, open access may not change the calculated impact of an article. This would therefore provide no benefit in the process of peer review. However, as you pointed out in both your post and in another article you authored (http://1.usa.gov/xtQ3PS), there is strong evidence that open access increases article readership.

    My point with that statement addressed a problem in our current scholarly publishing system and not in peer review, which was muddied by my wording above. Discussions of open access would require another post entirely, so I will leave the topic as-is. Thank you for bringing this up. It is an important discussion that hits home for many authors.

  3. Report this comment

    Robert Judson said:

    Thanks for the great article, Josh. It was very thought provoking and you did a nice job highlighting the valid concerns with the current peer review system.

    It seems to me that one of the biggest dangers when using any “untested” system of reward or currency (in the case you describe, publications being the currency of the academic sciences), is the potential to generate incentives that are misaligned with the purpose or mission of the players involved.

    As your article describes, there is concern that in the current peer review system, “high impact publication” may not necessarily be equivalent to “research conducted with integrity”, but rather that many flaws and biases in the system have disconnected these two purposes. If this is true, and there really is a disconnection here, then we have generated a highly competitive system that is no longer optimized for encouraging “research conducted with integrity”, but is rather selecting for “high impact publication”, which might represent a different or only partially overlapping set of traits. Interesting to think about, and perhaps a little concerning.

    In regard to your comment: “Post-publication review would be an excellent supplement to the current review system to improve the rate of error correction in publications”: Recently, a platform launched for archiving and organizing post-publication peer-to-peer review at journallab.org. The system allows for both anonymous and non-anonymous commentary. It will be interesting to see if this platform actually improves the rate of error correction, as you suggest or, further, could be used to “test” the current system of peer review.

  4. Report this comment

    Neil McCarthy said:

    Really interesting thoughts. Tend to agree that although there may be some problems with peer review, it is the best system that we have. I would suggest that one of the key issues is the English language bias in the review process, which can make it more difficult for scientist with English as a second language to be as successful in the peer review process.

Comments are closed.