In science it pays to have friends. Peer reviewers picked by the authors of a manuscript tend to provide more favourable feedback than scientists selected by the journal’s editors. That’s the unsurprising conclusion of an analysis of more than 500 manuscripts.
The suggestion that friends are kinder than strangers in peer review has been made before; analyses of the Journal of Clinical Investigation, the Journal of Investigative Dermatology and others have concluded as much. The issue also came up last year, when a group of stem cell researchers complained that overly chummy peer review was letting in inferior papers and delaying publication of better work.
But the new paper, published in PLoS-ONE, examines a journal with an open peer review system that might be expected to discourage such biases, says Lutz Bornmann, at the Office of Research Analysis and Foresight of the Max Planck Society in Munich.
He and his colleague Hans-Dieter Daniel, at the University of Zurich, Switzerland, focused on the journal Atmospheric Chemistry and Physics. The journal, launched in 2001, uses a semi-open peer review system, publishing the vast majority of articles it receives in a journal called Atmospheric Chemistry and Physics Disucussions.
There, the manuscript is reviewed anonymously by scientists recommended by the journal editors or the paper’s authors. Anyone else interested can post a review, as well. The authors then resubmit their paper for publication in Atmospheric Chemistry and Physics. Of the manuscripts submitted between 2001 and 2006, 86% were published in both journals, while 7% only made it to the discussions journal and another 7% didn’t even make it that far.
With access to reviews and information about who wrote them, Bornmann and Daniel found some signs of bias. On papers where there was disagreement among both kinds of reviewers, those recommended by the author were more likely to provide favourable feedback and accept a paper than the editor-recommended reviewer. And, after accounting for the quality of the papers via the citations that the papers would go on to rack up, Bornmann and Daniel still found a reviewer bias – small, but still there.
It’s not hard to come up with explanations for such patterns. “The danger is really that an author suggested their best friends,” Bornmann says. Alternatively – and more charitably – the reviewers selected by authors could in a better position to know a good result in their field when they see one, compared to those selected by journal editors, he says.
If indeed there a friend bias exists, Bornmann thinks journals should try to minimize it, without limiting their pools of peer reviewers. One option would be to direct author-submitted peer reviewers to other, similar papers.
Authors might also take their own initiative. When Bornmann reviews papers, he says he ignores the author’s name and institutional affiliation and concentrates on the paper and its data. Yet he admits that he has not tested whether he, too, succumbs to any bias.
“I haven’t done my own review,” Bornmann says. “I should.”
Image: photo by nieske via Flickr under Creative Commons
Report this comment
Its great to see that these biases are finally coming to light and it is being recognized as a potent threat to the fairness and quality that one expects in modern Science. I would also think that such biases would be reduced if journals adopted a double blind method of reviewing papers. Such that even if “friends” do end up reviewing papers – they will never be sure of who the authors are.
Report this comment
People in the same field tend to see the same group of colleagues at conferences and read papers among themselves all the time, so that even if a name is removed from the paper under review, the reviewer can often recognize (consciously or not) whose paper it is. My boss once recognized the writer of a one-page review of a grant by the style of her writing. Your peers are in the field, so they know your work well enough to judge it fairly, hopefully. But they also know you, and can recognize your work from a mile away, names applied or not. This implies that bias (positive or negative) is going to happen. Being kind is a good idea.
Report this comment
Academic journals desire to identify and publish those scientific reports that will have the greatest impact in their fields, and they also wish to avoid the embarrassment from being associated with inferior work. However, the major challenge with peer-review is that it is extremely diverse in quality with relatively little accountability. Moreover, direct competitors can have strong negative bias towards a scientific manuscript just as much as author recommended colleagues will have a favorable disposition. At the end of the day, however, it is really up the to general scientific community to accept or disregard the validity of the data and conclusions in a paper.
In the 21st century, with the advent of Internet publishing and the irrelevance of page constraints, it’s time to try something different. I suggest that that peer-review should not be completely anonymous. The referees should be selected in mutual agreement with the journal and authors from a short list provided by the journal. For the first stage of the review, it should not be exactly clear to the authors who reviewed their paper in case it is flatly rejected. However, once a manuscript is accepted for publication, the referees’ names should be identified right after the list of authors on the published paper. In this way, the referees receive recognition for their efforts and also accept responsibility for the quality of the paper’s content.
Ideally after publication, it should also be possible for readers to add their comments to the end of a manuscript if they can make meaningful and helpful contributions. Such a process should result in better quality publications that can also generate stimulating discussions with interested parties in the field.
Report this comment
Open review – when the author and everyone else can see who you are and exactly what you’ve said about their work – will encourage, rather than discourage chummy reviews. I can be much more honest and critical of a friend’s work in private to an editor (or even in private to the friend) than in public on the web.