Action Potential

Double-blind peer review?

Here’s a question I’ve heard a few times: why don’t we at NN, or any of the Nature journals, strip the author names off a manuscript before sending it out to peer review? This process, where not only the referees remain anonymous to the authors, but where also the authors might remain anonymous to the referees, is termed “double-blind peer review”, and is practiced by some specialized biomedical journals. Recently, a group of young scientists published a plea to adopt “DBPR. A 1990 study published in JAMA concluded that DBPR improved the outcome of peer review; nevertheless JAMA itself has not adopted DBPR.

When confronted with the question, I tend to reply that DBPR simply wouldn’t work for NN, because most reviewers would find it easy to guess the authors of a manuscript before them. After all, before you submit a paper to us, you have typically already presented a good part of the data to the community in the form of meeting posters, invited talks, etc. Also, authors tend to extensively cite their own previous work… My dialogue partners usually agree with these arguments, but still – DBPR would be easy to implement, so why not go ahead, even if it “worked” only some of the time?

Good question… I’d be very interested in hearing your opinions on this subject!

Comments

  1. Report this comment

    Bill Hooker said:

    Authors can also often figure out who the reviewers were, even in large and diverse fields like HIV and cancer biology. The pool of directly-relevant reviewers is not all that large and frequently includes one’s “competitors” (I hate that idea, but that’s how it’s typically phrased). Moreover, there is a group of “usual suspects” (e.g. those known to reply promptly, favourites of editors the world over!), and scientists tend to have distinctive pet issues and writing styles.

    But for all that, journals still keep their reviews “anonymous”, so I tend to agree — why not extend the same favour, which can hardly do harm and might do some good, to the authors?

  2. Report this comment

    Anonymous said:

    I read with great interest your comments on DBPR. My feeling is that DBPR doesn’t have to be perfect, as reviewers’ job is not to identify authors’ name, but to judge how a manuscript scientifically sounds. Plus, there would be no substantial demerit to guess the authors as it is open in most journals.

    Several reviewers (scientists) sometimes find it reluctant to make frank and critical comments on a paper where a Nobel laureate, for example, is one of the authors of a manuscript. I even know several authors who know this fact and intend to abuse it, namely just adding his/her credit with very minor scientific contribution. DBPR for instance becomes powerful to fairly judge scientific contsnts of such manuscripts. Needless to say, it is, in addition, so much incentive for young researchers as in your link to the plea (and easy to carry it out).

    So, let’s stop discussing how it could fail, but look forward to reviewer’s face having unexpected authors in press!

  3. Report this comment

    JB said:

    A postdoc in a lab at a neighboring school got her paper rejected for a very strange reason, and we think it’s because papers from Korea are being almost overzealously screened in the wake of the stem cell scandal. She’s a behavioral neuroscientist at a completely different school, nothing to do with Hwang or stem cells or SNU, and yet she may be being judged for something beyond her data and method. If anything, DBPR would assuage such fears concerning racial prejudice.

  4. Report this comment

    AMN said:

    I didn’t realize that the referee system wasn’t double-blind; I can’t see how peer review could be fair without it.

  5. Report this comment

    Jeffery Yang said:

    I am in the neuroscience basic research. Well, I have been reviewing lots of papers recently. Say, 20+ over the last 18 months or so. 3/4 of them come from my closely related field, thus I know who might be the authors, even you just showed me the figures, not even with any text. For those not from my immediate field, I have to say, I cannot guess no matter what.

    Will the author change my view? 99% no. Means I generally judge the work by the work itself, nothing personal. The 1% I left out is when I really don’t trust a person, and his work of course. This has never happened to me, since most of my review requests come from journals like Nature, Neuron, J Neurosci. The editors probably alrready screened them extensively.

    So I will say since DBPR does not hurt the current system, some journal should try it.

  6. Report this comment

    Anonymous said:

    One more thing. If you initiate the DBPR, it should be systematic; including all titles in NPG at least. It would be embarassing if reviewers whom NN used had reviewed the MS itself for another journal which had opened the authors’ names as a default. Lucky enough, papers sent to the journals in NPG are mainly the first submission, I suppose.

  7. Report this comment

    Anonymous said:

    Even if it failed, it won’t hurt. So why not try it.

  8. Report this comment

    Jonathon said:

    NN should adopt this policy. Or at least give it a try. Maybe all life science journals should take a crack at this method for peer review. I see no harm.

  9. Report this comment

    MM said:

    It appears that implementing DBPR is a no-brainer. “DBPR would be easy to implement, so why not go ahead.” I agree. There seem to be no drawbacks. The only drawback I could imagine would be the false idea that one’s identity is concealed to the reviewers in the cases where the reviewers think they can identify the authors by looking at their references or simply by their familiarity with the field (as mentioned above). Even still, this is not necessarily a drawback because there is no confirmation that the reviewer’s suspicions are correct until publication time. As far as initiating DBPR systematically, it is just unnecessary. Starting one at a time is just fine. DBPR may not be perfect, but at least it will contribute to a more objective analysis of the scientific merit of each manuscript.

  10. Report this comment

    Theo Mantamadiotis said:

    I would argue that if DBPR is not implemented, then what about a completely transparent review process where reviewers stand by their comments and reveal their identity or at the very least affiliation.

  11. Report this comment

    Andrew Matus said:

    Having spent many years reviewing and being reviewed I have positive feelings about dbpr. In the past I have been accused of unfairly reviewing papers I had never seen and, mutatis mutandis, on a couple of occasions when I was certain a particular colleagues had reviewed a paper of mine and asked them it turned out I was wrong. Thus my admittedly anecdotal evidence suggests it’s really diffcult to guess the identity of an author or reviewer. Dbpr might even encourage authors to more balanced in their citations so why not try it or at least make it an author option?

  12. Report this comment

    Anonymous said:

    Double blind review is a good idea. Beyond the obvious arguments relating to such a procedure being more objective, the idea of a “veil of ignorance” from the theory of justice (due to philosopher John Rawls) comes to mind. The original idea and context is different, but a basic notion that may still apply is to remove “morally arbitrary” identifying information in order to provide fairer judgements …

  13. Report this comment

    VM said:

    I would agree that DBPR is a step in the right direction. However, I would contend that it does not address the most important juncture at which an author’s name or reputation influences decisions on manuscripts. At high profile journals the most difficult and frustrating obstacle is often getting the manuscript past editors and into the hands of reviewers. Reviewers at least have to write down their concerns, comments and complaints which provide a concrete basis for persuasion, argument and further experiment. Scientists can have their opinions swayed in such ways even if their initial decision is to reject. So, if an author can get reviews and there isn’t a fundamental flaw in the data then they have an excellent chance of eventually addressing the reviewers concerns and getting the manuscript published. On the other hand, when an editor simply says “this isn’t of sufficient general interest for our journal” the author is stymied. My experience would suggest that editors are much more reluctant to refuse sending a paper to reviewers when the authors are “famous”. I’d rather have blinded editors than blinded reviewers though I realize this would not be easy to implement as a practical matter.

  14. Report this comment

    Attila Köfalvi said:

    I would reverse the question. Sometimes it is indeed easy to guess who the reviewers were. Therefore what if the identity of the referees was uncovered after the acceptance or the refusal of the manuscript? What if the published paper had one more sentence above the title: reviewed by Dr. XY and Dr. WZ? I know a great scientist who is very stubborn to include his name into his referee comments, which of course always give extra work to the editors to delete. He is a rare example. But in fact, the big shots’ work is often less criticized compared to that of a less well-known scientist (especially from outside the US – however, there is life outside the US!). As for me, I would never mind to uncover my identity to the authors, even if my opinion on the manuscript is unpleasant. It is because I think I am as fair as I can, and for me, the scientific content is the important. This might help increasing the referees’ responsibility and the value of peer-reviewed science.

    I also agree with the previous post. The biggest step is to get through the editors. It sometimes happens that the corresponding author calls the editor to discuss the fate of the manuscript. But I also agree with that publishing and maintaining a constant high-quality journal need to exclude all risk factors, i.e. to prefer papers from widely respected labs to those of less well known.

  15. Report this comment

    ueli said:

    In some fields of computer science, highly competitive conferences like NIPS, CVPR or ICCV use double-blind review; I have both submitted to and reviewed for such conferences and have had a very positive experience.

    Even if I can “guess” who the author or reviewer is, it is still a guess and thus their is uncertainty.

    I would very much favor a double-blind peer review process.

  16. Report this comment

    David Dobbs said:

    I side with Horace Judson Freeland, who in “The Great Betrayal: Fraud in Science,” argues that it’d be better to make public the names of the reviewers, so they are answerable not only to the author but to the public for their work. That might tend to create more care and be a healthier correction to this obvious imbalance.