Peer-to-Peer

Working double-blind [corrected]

This is the text of an editorial published in Nature in the 7 February issue (Nature 451; 605-606; 2008) . We, the journal editors, welcome your comments and suggestions.

Corrected 4 June 2008: Nature 453, 711 (2008).

Please see end of this post for text of correction correction to this editorial*

Should there be author anonymity in peer review?

Double-blind peer review, in which both authors and referees are anonymous, is apparently much revered, if not much practised. The Publishing Research Consortium (PRC) has assessed attitudes towards peer review among 3,000 academics in an international survey across the sciences and humanities. The results, released last month, strongly affirm the value of peer review [See earlier posting on Peer to Peer.]. They also highlight that 71% have confidence in double-blind peer review and that 56% prefer it to other forms of review. Support is highest with those who have experienced it (the humanities and social sciences) or where it is perceived to do the most good (among female authors). The least enthusiastic group is editors. So is it time for editors, and those at Nature in particular, to reconsider their position?

If referees know the authors’ identities, it may leave the latter vulnerable to biases about them or their previous work, their gender, their nationality or their being new to an area of research. But the PRC survey supports the contention of Nature and others that identifying authors stimulates referees to ask appropriate questions (for example, differentiating between a muddy technical explanation and poor experimental technique). Knowing author identities also makes it easier to compare the new manuscript with the authors’ previously published work, to ensure that a true advance is being reported. And knowing rather than guessing the identities of authors encourages reviewers to raise potential conflicts of interest to the editors.

Is there evidence that double-blind peer review presents a better alternative? It would do so if it generated more constructive comments in the minds of editors and authors, or if the identity of authors were truly protected, or if biases were reduced. So far, the jury is out. Although at least one study in the biomedical literature has suggested that double-blind peer review increases the quality of reviews, a larger study of seven medical journals (refs 2, 3) indicated that neither authors nor editors found significant difference in the quality of comments when both referees and authors were blinded. Referees could identify at least one of the authors on about 40% of the papers, undermining the raison d’être for double-blinding. The editors at the Public Library of Science abandoned double-blind peer review because too few requested it and authors were too readily identified.

The one bright light in favour of double-blind peer review is the measured reduction in bias against authors with female first names (shown in numerous studies, such as ref. 4 [previously discussed on Peer to Peer]). This suggests that authors submitting papers to traditionally minded journals should include the given names of authors only on the final, published version.

The double-blind approach is predicated on a culture in which manuscripts-in-progress are kept secret. This is true for the most part in the life sciences. But some physical sciences, such as high-energy physics, share preprints extensively through arXiv, an online repository. Thus, double-blind peer review is at odds with another ‘force for good’ in the academic world: the open sharing of information. The PRC survey found that highly competitive fields (such as neuroscience) or those with larger commercial or applied interests (such as materials science and chemical engineering) were the most enthusiastic about double-blinding, whereas fields with more of a tradition for openness (astronomy and mathematics) were decidedly less supportive.

Where does this leave journals? Editors have the responsibility to provide a neutral bridge between referees and authors and so may help to better shield authors from bias. Easily said! The evidence of the PRC survey suggests little faith in that impartiality, but editors — certainly at Nature and its related journals — take that responsibility seriously.

Nature’s policies over the years have generally moved towards greater transparency. Coupling that with the lack of evidence that double-anonymity is beneficial makes this journal resistant to adopting it as the default refereeing policy any time soon. But many of our readers are referees as well as authors. We welcome their views on author anonymity from both vantage points.

1. Publishing Research Consortium Peer Review in Scholarly Journals (Mark Ware Consulting, Bristol, 2008).

2. Justice, A. C. et al. J. Am. Med. Assoc. 280, 240–242 (1998).

3. Cho, M. K. et al. J. Am. Med. Assoc. 280, 243–245 (1998).

4. Budden, A. E. et al. Trends Ecol. Evol. 23, 4–6 (2008).

The Nature journals’ peer-review policies are described at our Author and Reviewers’ website, where you can also find links to our editorials on the topic, all free to access.

CORRECTION to the editorial:

The Editorial ‘Working double-blind’ (Nature 451, 605–606; 2008) referred to a study1 that found more female first-author papers were published using a double-blind, rather than a single-blind, peer-review system. The data reported in ref. 1 have now been re-examined2. The conclusion of ref. 1, that Behavioral Ecology published more papers with female first authors after switching to a double-blind peer-review system, is not in dispute. However, ref. 2 reports that other similar ecology journals that have single-blind peer-review systems also increased in female first-author papers over the same time period. After re-examining the analyses, Nature has concluded that ref. 1 can no longer be said to offer compelling evidence of a role for gender bias in single-blind peer review. In addition, upon closer examination of the papers listed in PubMed on gender bias and peer review, we cannot find other strong studies that support this claim. Thus, we no longer stand by the statement in the fourth paragraph of the Editorial, that double-blind peer review reduces bias against authors with female first names.

References

Budden, A. E. et al . Trends Ecol. Evol. 23, 4–6 (2008).

Webb, T. J. , O’Hara, B. & Freckleton, R. P. Trends Ecol. Evol. doi:10.1016/j.tree.2008.03.003 (2008).

Comments

  1. Report this comment

    David Lane said:

    I work in a field (replicon dynamics in microbes)in which the cooperation:competition ratio is, I would guess, relatively high. Subject, technique, citation habits and writing style would subvert >80% of attempts to conceal author identity, and I estimate the rate of correct referee guesses at about 50%. I doubt if double-blinding would be worth the candle in such a situation.

  2. Report this comment

    Nina Papavasiliou said:

    Peer review can be more or less biased depending on field. I “live” in a field where it is indeed extremely biased but not against sex – rather there is direct bias (both by editors and also by reviewers) against junior faculty combined with a relatively free ride for senior faculty.Of course, this is anecdotal, but a study on the issue broken down by field will, I think, be far more informative.

    Many junior faculty in my field have direct experience of this bias in peer review. How else would you call it when a paper with a set of data but junior authors is rejected by the journal (e.g. Nature) without review, when the IDENTICAL results (authored by very senior authors) are reviewed by the same Editor and published with much fanfare? (This has happened to me twice, and if you are interested I can substantiate with exact publications, dates etc).

    Is double blind (or even open) review a cure-all? Not necessarily, and there are definitely good reasons to argue for/against both options (including but not limited to the reasons you mention). But here’s another thought, that comes from being on NIH review panels: What if review is changed by having each reviewer identity and their comments blogged to one another before a decision on the paper is reached? That will a) stop reviewers from making silly comments (from asking for ridiculous experiments to killing the paper with a couple of unsubstantiated lines like “I see nothing new here” – as they will not be hiding behind anonymity in their reviewer group) and b) may even increase the quality of papers (as reviewers could, in real time, convince one another and reach consensus prior to a decision).

    As it is, many good papers from junior labs are not published and lots of rather thin papers from senior labs are. This lopsided system does a disservice to Science both directly (in terms of quality of what’s published) and indirectly (in terms of not showcasing the work of good junior investigators).

  3. Report this comment

    Sridar Chittur said:

    I think a double blinded peer review offers a truly unbiased platform. This allows for the research to be evaluated purely on the basis of the presented findings rather than the reputation of the author. As such, novel findings from lesser known PIs may find more honest review without any pre-existing biases on part of the reviewers. Additionaly, reviewers may subject manuscripts from well known labs to the same scrutiny preventing errors from being published.

  4. Report this comment

    JZ said:

    It’s great if the journal editors could provide two options for the authors: double-blind peer review or not, although there will be more workload to the editors.

  5. Report this comment

    Dinesh said:

    I strongly believe that double blinding the review will take care of a lot of bias not only in relation to gender, experience in research (junior/senior researcher) but also the bias of the country.Most papers are simply rejected just because they are from third world counties and the science done is not taken seriously. This will be a great thing to do. There should be a “complete double blind” review process so that no meta information about the authors identity is known to the reviewers. Also, the authors should be told a priori not to use phrases like “our previous work (ref) shows that”, " in continuation to our previous work", etc. This will increase the credibility of the work done world wide and its acceptance to the scientific community in general. This should not be made optional just to collect data demonstrating that it does not work and revert back.

  6. Report this comment

    L Prugh said:

    This editorial states: “The one bright light in favour of double-blind peer review is the measured reduction in bias against authors with female first names (shown in numerous studies, such as ref. 4 [previously discussed on Peer to Peer]).” This statement is soon followed by: “…the lack of evidence that double-anonymity is beneficial makes this journal resistant to adopting it as the default refereeing policy any time soon.” Apparently the editors of Nature do not feel that reducing the substantial and well-documented bias against female authors is beneficial. I find this to be rather depressing.

  7. Report this comment

    Alejandro Iglesias said:

    Double-blinded peer review can be very helpful for female authors, non-English speakers and authors from developing countries. All of them, potential contributors to their fields, can bring their own experiencies and research, that otherwise will not be in high impact journals. Although, knowing the authors can help to address the comments and assess progress, it should be possible to do it without knowing their identities. Moreover, that will be the real proof of an unbiased review. Somehow, not doing double-blind peer review might carry a risk for discrimination while at the same time playing a negative role in the diffusion of good science.

  8. Report this comment

    Loren Byrne said:

    I agree with Nina’s post about the potential for bias against junior scientists. Based on my personal experiences, I feel that there may be some inherent bias against accepting papers from young scientists who are trying to establish a career and do not yet have widespread name-recognition. Could it be that not recognizing a lead author’s name (or perhaps other authors also) leads reviewers and editors to be less accepting of the new work and ideas? After one of my novel papers was rejected despite receiving overall positive review, I wondered if my paper would have been accepted if it had a different name on it. While double-blind review may not be a cure-all, it could certainly avoid some instances of sub-conscious bias in instances like this. (And it should be noted, the question of reviewers guessing who the author is in a double-blind review may be moot in many cases when the paper is by a junior scientist who is unlikely to be known or has not published extensively enough to allow guessing.)

  9. Report this comment

    Matt Chew said:

    Anonymous peer review is not peer review is not peer review at all; anonymous authorship seems to offer few remedies to that situation.In highly specialized fields, most authors know who their likeliest reviewers are. Supposedly anonymous reviewers often reveal themselves idiomatically to those familiar with their work, and word often “gets around” anyway.

    Reviewers should be willing to stand up and say what they have to say in an open and transparent forum. That may frighten away those who cannot maintain a civil discourse, but it will also facilitate the solution to another problem: peer review is rarely considered on par with publishing,professional service or teaching in performance or retention reviews, but it is a critical and time-consuming part of science. Identified reviewers can more easily document and take credit for their contributions.

    Anonymous authorship will not improve the quality of submitted papers. Identifying reviewers will improve the quality of reviews, and reduce the level of conspiracy theorizing and suspicion that attends anonymous panning.

  10. Report this comment

    Samy Sakkal said:

    As a junior scientist I like the idea of anonymous peer review, in principle, because at the end of the day, it should be all about the science. However this also creates issues too because not all scientists actually perform “good science”. This is where knowing an authors identity/pedigree/reputation can be useful since different institutions/labs have different standards. Thus I would be more inclined to accept a publication from a lab that has never had a retraction or committed fraud compared with a lab that has been guilty of such, esp on several occasions. Granted, whilst even some of the greatest scientists have retracted at least one paper in their lifetime, they are generally rare because of the quality control measures that the lab head instills. Since their appears to be compelling reasons on both sides, perhaps journals could consider doing both; that is since manuscripts are sent to two reviewers, one has the authors removed and the other where authors are listed. Perhaps Nature could try this, and if their are discrepancies, like all manuscripts, it goes to a third reviewer, which would be also blind, but privy to both reviewers comments.

  11. Report this comment

    Thomas Sandmann said:

    Nature considers itself “resistant” to adopting double-blinded reviews a default refereeing process. Nevertheless, several arguments put forward against this flavor of peer review in this editorial can also be interpreted in a positive way, e.g.:

    “[…]neither authors nor editors found significant difference in the quality of comments”

    Doesn’t this indicate that double-blinded peer-review does produces reviews of equally high quality ?

    “Referees could identify at least one of the authors on about 40% of the papers[…]”

    In 60% – the majority – of cases referees could not identify any author. And, importantly, even the 40% of cases where at least one author was guessed correctly, the reviewers could be anything but sure that they were right.

  12. Report this comment

    Kostas Alexandridis said:

    A number of scientific journals have replaced the double-blind peer review process with a triple peer review process. As both an author and a peer reviewer in those journals, I found the latter process more transparent and objective than the former. The double-blind peer review process, whilst useful, it tends not to serve the purpose for which it has been intended, that is to ensure anonymity of the authors. My experience shows that in most of the submitted manuscripts, one (or more) of the authors can be identified or directly/indirectly inferred by the manuscript. Personally I do not believe that revealing the identity of the author(s) affects the quality of the reviews received by peers (yet, I can see how in principle some instances can occur when a larger bias exist). The introduction of a third reviewer can compensate for such problems as it moves the peer review process towards a more objective assessment pathway. Furthermore, often the review process can benefit by assessing the potential mind-set and scientific background of the author(s) by improving the quality of the comments and direct the authors on more informed ways of improving the quality of the manuscript. The latter is especially true in science directions where the multi- and cross-disciplinarity of research present increased risks on covering a wider range of scientific literature and body of work. Peer reviewers that have a wider range of expertise in different disciplines related to the research under review can then provide more useful comments and directions for the authors of the manuscript. Finally, in those situations, the usefulness of adding a third peer reviewer also helps on ensuring a more holistic and inclusive peer review process.

  13. Report this comment

    Abhay Sharma said:

    If “identifying authors stimulates referees to ask appropriate questions (for example, differentiating between a muddy technical explanation and poor experimental technique). Knowing author identities also makes it easier to compare the new manuscript with the authors’ previously published work, to ensure that a true advance is being reported …” is true then, in the same way, knowing the reviewers’ identity will also be productive for the authors, i.e., it will enable the latter to answer reviewers’ comments more effectively in light of their publications! In brief, double-blind review is ideal. The most important point in this favor being “if referees know the authors’ identities, it may leave the latter vulnerable to biases about them or their previous work, … , their nationality or their being new to an area of research”.

  14. Report this comment

    Maxine said:

    Maxine responds to L. Prugh:

    You write “Apparently the editors of Nature do not feel that reducing the substantial and well-documented bias against female authors is beneficial. I find this to be rather depressing.”

    The “well documented” bias is not documented at all for Nature, but has been claimed in small studies on particular journals, not Nature journals. At Nature, the average number of authors per paper is five. Many authors use initials only, or have gender-neutral first names. We do not collect gender data at submission as our policy is that we require authors to provide information only if it is relevant to their submission (address and so on). [Would authors like to be asked their gender at submission, I wonder?]

    Therefore, in the vast majority of cases, the authors of a paper are a mix of genders, and in very many cases, the authors in the list are of unknown gender to the editor because only initials are provided, or because first names are gender-indistinguishable.

    Another point to note is that more than half of the manuscript editors at Nature (those who select referees) are women — see journal online masthead (http://www.nature.com/nature/about ) for biographies of the editors. Two of the three senior editors who set the journal editorial polices are women.

    At the Nature journals we are proud of our independence: we do not have an editorial board of a few scientists, for example. We certainly do not agree with L. Prugh that Nature is in any sense biased for or against any particular group or person. We are interested in one main goal: publishing excellent science, irrespective of who performs it.

  15. Report this comment

    Bob O'H said:

    The one bright light in favour of double-blind peer review is the measured reduction in bias against authors with female first names (shown in numerous studies, such as ref. 4 [previously discussed on Peer to Peer]).

    The data given in Budden et al. aren’t so clear, as I argued on my own blog.

    I’m also curious about the “numerous other studies” – I can’t find any in Web of knowledge (searching for “gender bias” AND “double-blind”). Any references?

    Bob

  16. Report this comment

    Erec Stebbins said:

    Science is a social endeavor, and scientists no less prone to social forces than anyone else. This means that the review process will contain many political as well as scientific aspects. Therefore, discrimination based on perceived gender, race, age, and so forth, if present in general in the population, will be present in scientific review. To question this is either naive about human nature, or willfully self-deceptive. Processes to minimize such bias should therefore be adopted.

    In addition, power structure, as in all human activities, insures that those with power seek to keep it, often through the abuse of the power that they have gained. This easily explains how the fledgling field of modern biology in the mid part of last century went from a youth dominated field to the much older spread seen now (just look at NIH money distribution vs. age, average age for first RO1, etc). One could argue that all these forces are detrimental to science.

    Double blind peer review would help to address these concerns, although certainly not eliminate them. Completely open peer review also has its advantages and disadvantages. However, both are superior, to my mind, to the current system that protects the establishment and allows, when present, bias to thoroughly taint a review process.

  17. Report this comment

    Maxine said:

    That is an excellent blog post and comment thread, Bob. Do the authors of the TREE study accept the reworked statistics? If so, it seems their conclusions may need modifying in light of the revised statistical uncertainties.

    C. Wenneras and A. Wold, Nepotism and sexism in peer-review, Nature 387, pp. 341–343 (1997) showed a clear gender bias in peer-review of grants, based on raw data obtained via the Swedish Freedom of Information Act (ref. 3 in the TREE paper).

    Although there are plenty of studies showing or claiming gender bias in various aspects of scientific evaluation, I am not aware of other studies showing hard data on journal peer-review, although there are plenty of surveys of the journal process that are not conclusive for various reasons (search Google Scholar with “peer review” and “gender bias”, for example). I’ll follow up on your question. Of course there are many, many studies showing gender bias in various aspects of scientific review and assessment, and in the profession generally (see for example /nautilus/2008/02/underrepresentation_of_women_i.html ).

  18. Report this comment

    Nina Papavasiliou said:

    Interesting thread…

    1) I would hazard that the cohort clamoring most for a change in the review process is the cohort which perceives themselves to be at a disadvantage – I have not heard senior colleagues complain about this for instance. Scientific experiments are done double blind to remove bias. Why not peer review?

    2) To L Prugh and Maxine: Having women editors does not mean that sexism does not exist – in fact it is often true that women judge other women far more harshly than men would. And even if the women Editors of Nature are absolutely gender neutral, can the same be said about Nature’s reviewers?

    •In a classic social psychology experiment, hundreds of subjects were asked to evaluate the same article, which differed only in whether it were supposedly written by a male, a female, or an author with a sexually ambiguous name. The essays attributed to the male authors received on average a significantly higher grade, from female raters as well as male raters (Paludi and Strayer 1985).

    (http://www.indiana.edu/~college/faculty/policy/collegepolicies/gender2005.shtml)

    * also check out http://dynamic.uoregon.edu/~jjf/chillyclimate.html#PeerReview

    I wonder whether, given this amount of evidence, sex (and other) biases can even remotely not taint peer review.

    3) isn’t it time for Nature, as the pre-eminent scientific journal, to actually conduct its own research on the matter (to look into how age and sex have affected manuscript decisions, ie to establish whether bias exists and in what form, and then to assess whether double blind review ameliorates that?)

  19. Report this comment

    delia said:

    I have witnessed slanted behaviour from several scientist peer-reviewers who were preoccupied that other people were studying the same thing, hence found all possible problems in papers in order to recommend rejection. I also know peer-reviewers who did not point out critical problems in a paper because the authors were “friends” or “politically useful”. Therefore I believe that authors and referees should be blind.

  20. Report this comment

    LC said:

    I really don’t understand what is this discussion about. It is clear that double blind is the way to do it as there is no logical reason why it should be single blind. Nobody here seems to consider the huge advantage in terms of faith in the refereeing process that young students would have. I can’t count how many young scientists were thrown off the academic career by the ugliness of the peer review system and of the perceived (and many times proven) bias.

    It is ridiculous people are even defending a system that is obviously wrong.

    I understand that it would remove power from the editors and this is why they are refraining from using it and also why some referees might take pleasure in knowing the names of the authors they have to referee… as it gives them a chance to vent their frustrations. How to justify otherwise all those referees which take the time to personally insult the authors?

    So, please, drop the philosophical and artificial motivation and let’s speak clearly.

    Single blind is damaging science and its credibility, especially in the eyes of young scientists like me and many like me and you are failing to recognize this.

  21. Report this comment

    Sebastian Fugmann said:

    The answer is simply “Nature should give it a try”.

    I have heard many arguments against and in favor of the double-blind approach, and as a true scientist I think the only way to resolve it is to perform a well-controlled experiment. Looking at somebody else’s experiment (in this case e.g. PLoS) is a good way to form a hypothesis but obviously not proving it.

    When reviewing a scientific paper it should be irrelevant whether the corresponding author is a nobel laureate or an unknown junior investigator, whether the lab is located at one of the most prestigious universities or at a college in the middle of nowhere, whether it is written by a male or a female scientist, … (this list could be extended forever). What really matters is the science that represents the core of each manuscript.

    Would the quality of the reviews suffer from not knowing (or guessing who the authors are)? Probably not – if it is a bad paper, it stays bad no matter whether it originates from an unknown or a famous lab – if it is an outstanding paper it would probably be published no matter who the authors are – but if it is one of those borderline manuscripts (and I think each and everyone of us has seen papers in Nature where we asked ourselves: why did this get published here and not in XYZ) I think our recommendation would be rather “Reject”. And thus this approach may actually improve the overall quality of the published papers.

    Is there anyone who would suffer the most from this scenario? The answer in my opinion is pretty clear – it is going to be those PIs who get the benefit of the doubt because of “whatever your most favorite reason is” …

    I talked to many of my junior faculty colleagues and none of them thought that the double-blind approach would actually harm their chances of getting published in Nature (the most negative outcome envisioned was that there won’t be any change). We are willing to take the risk that there might be serious disadvantages for us in this approach that we don’t know about.

    Let’s just give this approach a fair chance for the benefit of science in general!

  22. Report this comment

    Marilyn Diaz said:

    The system is biased especially against junior scientists. Double-blind reviews would help alleviate this problem, since even if people suspect who the authors may be, they won’t know for sure. The case made against double-blind reviews seems weak, and I don’t think it outweighs the benefits. Perhaps, Nina’s suggestion for the identity of the reviewers to be known among the reviewers would be a step closer to minimizing bias. One thing is for sure: the current system is far from perfect and something must change to preserve the credibility of the peer review process.

  23. Report this comment

    Gong Chen said:

    Double blind is ideal!

    I recommend every journal adopt double blind policy.

    We all know that blind analysis of our experimental data gives most accurate results. The same applies to review process.

    The editors can send the abstract and figures with legends without authors’ names to potential reviewers to solicit review. A good reviewer will like to review an exciting finding without knowing who made the discovery.

    Who makes the discovery is irrelevant during the review process!

    Double blind will reduce the bias toward newcomers to the field or favorable bias toward the leaders in the field.

    Everyone should be equal in the review process!

    I just received a manuscript for review, and I don’t know the authors’ names. Even though I can guess one author of the paper, it is completely a different feeling from seeing the authors’ names. After all, it is my guess. I could guess it wrong. Just like we all try to guess who might have reviewed our papers, we can never be sure. Without knowing exactly who the authors and their collaborators are, we can more concentrate on the experimental data itself!

    If every journal, especially Nature Publishing Groups adopt the double blind policy, it will be a great relief for the authors, the reviewers, and the editors.

    We should not care about who did the work. Rather, we should care about whether it is an excellent work or not!

  24. Report this comment

    Jason Snyder said:

    I think double blinded reviewing may be as good or better than the current system. Yet, I definitely see the benefit of knowing the identity of the authors so a submission can be interpreted in light of the authors’ previous work. Personally, I worry that, if both the authors and reviewers are made known to each other, decisions will be taken personally and may only make things worse. What I like best is what Nina suggested, above, or something like it where reviewers can share comments amongst each other before coming to a decision. Or, if they choose not to interact, at least have their comments available for the other reviewers to see prior to a decision being made. This may make reviewers think twice before making their comments.

  25. Report this comment

    Bob O'H said:

    Maxine – thanks for your comment. I guess it’s worth emphasising that I’m not arguing that there is no effect of double blinding, only that the numbers don’t show it.

    The only thing I’ve heard from Budden et al. is the comment on my blog, but I would be surprised if there isn’t anything in TREE about it in the near future.

    On double blinding, I’m not convinced that it will solve the problems. The charge against single blinding is that it biases against scientists that are less well known. If the identity of a substantial minority of authors (or research groups) can be worked out in double blinding, then I would assume that these are the better-known workers, and hence they should still have a bias in their favour. The overall effect may therefore be to shift the bias, so that middle-ranked authors aren’t favoured so much. In other words, it just shifts the point where a bias starts to operate.

    If this is true (and it is only speculation), then I think it would suggest that biases will be almost the same for journals with a high rejection rate (e.g. Nature). I might have to fiddle with some models of this tomorrow.

  26. Report this comment

    Maxine said:

    “LC” writes “there is no logical reason why it should be single blind.”

    This is not a correct statement. There are plenty of logical reasons.

    See http://www.nature.com/authors/editorial_policies/peer_review.html, which explains the Nature journals’ peer review system and why we think it is the best system. This page includes links to several free-access editorials in the Nature journals on the topic.

    I also recommend the article by Charles Jennings in the Nature peer-review debate, archived on this blog.

    We do welcome everyone’s views, and will consider them all carefully, but if people wish to make assertions it would be very helpful to us if they were supported by some evidence. For example JAMA and other journals have conducted objective studies of bias in their peer-review systems. Are the commenters in this thread who suggest such studies aware of this literature?

  27. Report this comment

    B. Kamenov said:

    The PRC report, this excellent blog, and the Nature editorial itself have clearly laid out the major advantages of a double-blind review system, compared to the single-blind one currently practiced by the nearly all journals. Yet, Nature remains “resistant to adopting it as the default refereeing policy any time soon”, echoing the attitude of many others. If there is such an unyielding opposition to what the majority of the scientific community feels is a better way to go (as the contents of this blog also make exceedingly clear), there must be some powerful reasons behind it. Let’s have a look at the reasons cited in the editorial:

    1) “identifying authors stimulates referees to ask appropriate questions (for example, differentiating between a muddy technical explanation and poor experimental technique)”

    2) “Knowing author identities also makes it easier to compare the new manuscript with the authors’ previously published work, to ensure that a true advance is being reported”

    3) “knowing rather than guessing the identities of authors encourages reviewers to raise potential conflicts of interest to the editors.”

    Are these reasons convincing? Not to this blogger. Reason 3) is clearly misguided – conflicts of interest arise from the subject matter of the manuscript, not from its authors; if the conflicts are of a personal nature, double-blinding can only reduce their occurrence. Reason 1) is exceedingly trivial. Instead of assuming that the explanation, rather than the technique, is wrong (always a risky assumption, even if a paper comes from a “good” lab), a reviewer should simply require the authors to clarify which is the case. Reason 2) has some substance, however, only a marginal one. In most cases the level of advance will depend on comparisons with a large body of work, from many different labs, and a good reviewer should be familiar with all of it. Indeed, a scientist is more likely to repeat the work of others than his own. Moreover, “novelty” assessments are usually made by editors, who know the identity of the authors and can examine their previous publications.

    Journals in the past have pointed to the burden of the additional work related to concealing the authors’ identities as a primary reason for their unwillingness to adopt double-blind review. With the universal adoption of electronic submission, however, this has become a non-issue and double-blind review is exactly as easy to achieve as single-blind. That leaves the case against double-blind review standing on its last leg – the claim that in many instances reviewers will anyway guess the authors’ names correctly. That is true. What is equally true, though, is that in many other cases reviewers will no longer be able to divine who the authors are, especially in dynamic, heavily populated fields of science. Moreover, reviewers are least likely to guess correctly the identity of junior newcomers to their field, who are probably the most disadvantaged group under the single-blind system. In addition, there are some very simple methods for minimizing the chance of author identification (e.g. prohibiting explicit self-citation), which may indeed be beneficial for the objectivity of scientific writing in general. At the end of the day, reviewers will sometimes know who the authors are, but very often they won’t. And that’s precisely the point. No one is trying to claim that double-blind review will eliminate all bias. The only claim is that bias will be reduced, and there is little doubt that this will happen. The studies that show no significant differences between the two systems date back to the 1990s, are confined to a limited number of papers in medical journals and are contradicted by other publications that do find a difference.

    In summary, double-blind review would require no extra effort or expense, will not have any appreciable negative effects, but might contribute substantially to enhancing the objectivity of the refereeing process. Even if this enhancement is not as significant as we may wish, at the very least double-blinding will boost the confidence of the research community in the good will of the journals to employ an optimal system of peer review, which takes into account the opinion and experience of scientists. So why not finally give it a serious try? Failure to do so may only strengthen the notion, rarely articulated but frequently suspected, that the true reason behind the refusal to grant author anonymity lies in pressure from established members of the research community. Such scientists, it is widely believed, find it easier to publish substandard work in top journals on the strength of their prior reputation and would therefore be opposed to a system where they can rely less on the benefits of their names. This is one reason why I believe that if double-blind review is to be tested in earnest it should be compulsory. Otherwise senior authors will opt against it and those who opt for it will often be assumed, rightly or not, to be junior, female or from outside the privileged circle of the rich, scientifically prominent nations and institutes. Hopefully, Nature and other journals don’t want such considerations to bias the evaluation of the science they publish.

  28. Report this comment

    LG said:

    Double blind review is better because it makes the involved parties concentrating more on the value of the research work itself, rather than other things. It maybe possible that the author identity may be guessed out and the refereeing process is less efficient, however it is definitely an improvement and reduces the “human factors” in publishing.

  29. Report this comment

    R.C.Sihag said:

    I agree that double-blinding peer review is the best method to avoid any bias against the author. Even on the peer review report ,the editors should encourage the authors to improve the quality through additions/deletions/alterations rather than straight rejections.

  30. Report this comment

    Glen Peters said:

    All the issues are solved if reviews are totally open. The authors and reviewers are all known to each other, but the critical part is that all readers can easily access the original article, reviewers comments, editors comments, and subsequent changes.

    The problem with today’s peer review system is that the reviewers and editors are totally unaccountable. The author does not get much say if a reviewer unfairly rejects work for non-scientific reasons (for example, a mainstream economist not agreeing philosophically with a non-mainstream economist).

    As for bias, if Reviewer XYZ is unfair, then the readers of that journal will perhaps not be so friendly to XYZ when they review their article. If XYZ is too soft on review, then the editor and readers need to let that be known.

    Perhaps it is different in other fields, but in my field I can usually tell who the author is without seeing their name, and 50% of the time I can tell who reviewed my articles. If the editor does not help resolve issues, then I usually contact the authors directly to discuss a point in more detail.

    My objective is to get good research out, then let the readers decide…

    What do we have to hide? I have rejected work of close colleagues! If my work is not up to scratch, I expect them to do the same to me…

  31. Report this comment

    Richard Blythe said:

    I generally concur with the view that authors’ identities can often be guessed. We recently submitted an article which (I think) will have the authors’ names removed; but the abstract, content and cited references will make it completely obvious who the authors are. There is no way for us, in this instance, to avoid this without being actively scientifically dishonest (viz, suppressing mention of earlier work, not exposing our methodology to scrutiny and so on). Of course this won’t be true of all papers.

    My experience with the standard review process is anecdotally suggestive that the higher the rejection rate of the journal, the lower the quality of the review. I have by now received from three major journals reviews containing unsubstantiated comments that have lead to the immediate rejection of what I have felt to be novel, interesting, thorough and reproducible science (often with no right to reply, apart from a torturous appeal process). Indeed, I have, as referee for a high-impact journal, seen a paper rejected on the basis of casual unscientific criticism when I recommended publication. I think there are serious issues with the single-blind system that stem from reviewer anonymity, no incentive to provide careful, considered reviews (good reviewers simply get asked to review ever more papers) and editors who not only return poor quality unscientific reviews to authors, but further base their decision on such reports.

    I wonder to what extent an open review process would resolve these problems.

  32. Report this comment

    Henry Gee said:

    As a Nature editor (and I stress that what follows is my own, personal view, not a reflection of Nature’s general policies) I am intrigued by the notion of double-blinding, as in the cause of perceived fairness it would make the process of peer-review even less transparent than it already is.

    So let’s try an alternative tack. At the moment, Nature’s referees remain anonymous unless the referee explicitly asks that their identity can be revealed. How would the community react were Nature to make the peer-review process utterly transparent by demanding that referees reveal their identities? I submit that this would expose far more bias against younger, female and other so-called ‘disadvantaged’ groups than people think there might be under the current single-blind system. It would also make it very hard for editors to solicit reports from younger researchers – non-tenured staff and postdocs, for example – who are often much closer to the action (and provide better-informed reports) than PI’s who spend less time at the bench.

    In general, I find that many misperceptions persist because scientists do not have a clear idea of what the role of an editor really is, and how journals such as Nature really work. Nature has internship programmes, and we do have locum positions for editors from time to time, all of which positions are advertised publicly.

    Maxine adds: much is explained about the editor’s role and the process in the “Ask the Editor” forum at Nature Network, http://network.nature.com , and on the “getting published in a Nature journal” pages (in eight languages) on the Author and Reviewers’ website at http://www.nature.com/authors

  33. Report this comment

    Erec Stebbins said:

    The main, repeated objection to double blind review I have seen voiced in this blog is the notion that the author identities can be guessed. While this may be an argument for open review vs. double blind, it can’t be for single blind vs double blind. In the latter case, if the authors are guessed, it will simply reduce to single blind. In small fields this will happen more often. In larger fields, I think this will happen much less frequently, with much less confidence, and certainly not in any field for young investigators without a prior “text fingerprint” to mark them (and who, as B. Kamenov noted in a very well argued and thorough post, are the ones most likely to benefit from double blind review). The argument is strongest for established senior investigators who will purposefully “mark” their manuscripts, if they should perceive an advantage in the revealing of their identities. The ingenuity of intelligent scientists to effectively conceal their identities when they perceive an advantage I think is being grossly underestimated! For all the major advantages in real and perceived objectivity, and the rather minor disadvantages, if any, it is startling that double blind review is not considered the common sense and obvious choice.

  34. Report this comment

    Erec Stebbins said:

    My problem with open review is that it risks opening junior, untenured researchers to retaliatory actions from those whose papers they have reviewed negatively. Human nature, and personal experience, is not encouraging in this area, and I believe strongly that anonymity is critical. I would support therefore a double blind system over an open system for this reason alone.

  35. Report this comment

    Maxine said:

    Maxine adds:

    Some points about double-blind/single blind review in response to some of those made here.

    1. In the single-blind system, journals such as Nature and the Nature journals allow authors to suggest peer-reviewers (these suggestions are often followed, as explained in the peer-review policies to which I’ve linked) and indeed to exclude named individuals or groups (within reason). It reads to me as if the commenters stating peer-reviewers are currently biased are unaware of these policies (to which I provided a link in an earlier comment in this thread).

    2. However “junior” an author, scientific research does not happen in a vacuum, and in 30 years of reading scientific research papers, I think I have never read one that does not reference the authors’ earlier work and build on it.

    3. In the single-blind system run by Nature: two, three and sometimes four peer-reviewers are used, depending on the expertise required to evaluate the manuscript properly. All these reviewers see each others’ reviews at each round of revision (there are at least two of these per submission). Why is this check/balance discounted by some of these commenters in favour of a peer-reviewer acting in splendid isolation? Why is one isolated judgement considered better than that of several independent scientists seeing each others’ views? (Do some commenters seriously think that the entire scientific community is in a consipracy?!)

    4. The editors of the journal choose the peer-reviewers for a submitted manuscript. They do a lot of work to find appropriate reviewers. Those in favour of double-blind might even prefer a triple-blind system — yet how could this operate? How,then, would the journal system differ from the author simply uploading their manuscript into a preprint server and leaving it at that?

    In sum, I am surprised to read so many commments asserting base motives to peer-reviwers and journals, without in most cases a shred of documentation, and without considering these and other factors. In my long experience as an editor, the vast majority of peer-reviewers provide constructive criticism of manuscripts, making many specific suggestions for improvements.

    Authors in general (in response to numerous surveys) feel their published manuscripts are improved by the peer-review process. A review that is a vague endorsement or a vague criticism on a submission is not considered by a Nature editor to be a useful review, as we ask for specific technical advice from our reviewers: in making a decision about publication we would use the other reviews for that manuscript or seek more specific advice, in which the reviewer substantiates her or his opinion (what we look for in a review is explained in some detail in our instructions to reviewers at the link I provided earlier in this discussion thread).

    Naturally it is disappointing to be told that your results were not considered as exciting as someone else’s, but the reviewers’ reports and the editor’s letter do provide specific reasons for these decisions. Nature is able to publish only a tiny fraction of the manuscripts submitted to it. We must be careful not to confuse “bias” with disappointment.

  36. Report this comment

    Erec Stebbins said:

    “1. In the single-blind system, journals such as Nature and the Nature journals allow authors to suggest peer-reviewers (these suggestions are often followed, as explained in the peer-review policies to which I’ve linked) and indeed to exclude named individuals or groups (within reason). It reads to me as if the commenters stating peer-reviewers are currently biased are unaware of these policies (to which I provided a link in an earlier comment in this thread).”

    This is of course a good policy, and one I am sure everyone is aware of, but it seems a patch to repair an already flawed system. And it does not insure that fair reviewers will be chosen. In a scientific experiment, double blind is preferred. While one wishes bias and mistakes would not creep into the experiment, the fact is, they do, and it is in the interests of the most objective outcome possible that double blind is employed.

    “2. However “junior” an author, scientific research does not happen in a vacuum, and in 30 years of reading scientific research papers, I think I have never read one that does not reference the authors’ earlier work and build on it.”

    This can vary extensively. Junior investigators who have moved into a different field (often to avoid competition with a former advisor, for example) will not be in a position to cite earlier work. Also, if the junior work is relevant and well published (meaning that the junior person was in fact very likely once part of a senior lab), it is not clear who is doing the citing – the senior or junior person. Finally, as mentioned, an author could easily cite related papers as well and construct the manuscript in such a fashion as to leave the author of the study very unclear. An enterprising junior person, for example (one of many), could seek to “mimic” the structure of a well-established senior author’s papers. Many variations on this exist.

    “3. In the single-blind system run by Nature: two, three and sometimes four peer-reviewers are used, depending on the expertise required to evaluate the manuscript properly. All these reviewers see each others’ reviews at each round of revision (there are at least two of these per submission). Why is this check/balance discounted by some of these commenters in favour of a peer-reviewer acting in splendid isolation? Why is one isolated judgement considered better than that of several independent scientists seeing each others’ views? (Do some commenters seriously think that the entire scientific community is in a consipracy?!)”

    Double blind review need not hide the reviewer comments from other reviewers. I don’t see the issue with that at all. As for conspiracy – is gender bias a “conspiracy”? Or other forms of discrimination? This word unfortunately marginalizes the discussion of what are serious social issues, very real ones.

    “4. The editors of the journal choose the peer-reviewers for a submitted manuscript. They do a lot of work to find appropriate reviewers. Those in favour of double-blind might even prefer a triple-blind system — yet how could this operate? How,then, would the journal system differ from the author simply uploading their manuscript into a preprint server and leaving it at that?”

    These are actually good questions. I don’t think anyone here is minimizing the work editors do. But even the best editors working within a flawed system cannot undo the inherent weakness of that system.

    “In sum, I am surprised to read so many commments asserting base motives to peer-reviwers and journals, without in most cases a shred of documentation, and without considering these and other factors.”

    As far as I can tell, there has not been any assertion toward the editors here directly, but a critique of the flaws in the single blind system. This is also a blog, not a court of law, so people are speaking from their own personal experiences, and have not necessarily constructed a case to be argued more formally (and few would feel comfortable in a pulblic form airing details that could lead them to have the fate of so many “whistle-blowers”). But that so many have spoken this way I think is telling.

    As for “base” motives – I think science is past the age of the ivory tower, pure-at-heart researcher. I’m not sure it ever existed anyway. There is big money, fame, and power in science now, as the increasing number of fraud cases clearly demonstrates. With that comes all the troubling aspects of any political system. Science as a whole has a goal of removing bias in order to seek objectivity, and throughout its history, despite many flawed systems, has consistently chosen a path to maximize objectivity. I think in this age one path we ought to take is to reassess our peer review system.

    “In my long experience as an editor, the vast majority of peer-reviewers provide constructive criticism of manuscripts, making many specific suggestions for improvements. Authors in general (in response to numerous surveys) feel their published manuscripts are improved by the peer-review process. “

    Again, I do not think that the critique is that peer review is bad, but that it is flawed in certain respects, and that there appear to be ways to restructure peer review to address those flaws.

    “Naturally it is disappointing to be told that your results were not considered as exciting as someone else’s, but the reviewers’ reports and the editor’s letter do provide specific reasons for these decisions. Nature is able to publish only a tiny fraction of the manuscripts submitted to it. We must be careful not to confuse “bias” with disappointment.”

    Nor should one assume there is not bias when there is disappointment. No doubt in the reviews of women faculty over the years, or decisions to hire, or award grants (all these subjects of study as you know) “objective” decisions were made, reasons usually given for those choices. These women were usually disappointed, and often felt there was bias. Was/is there? If there were a method to remove potential bias against a person, should it not be implemented, even if there happens to be no bias and it was “all in their heads”?

    Bias and society are complicated and often deeply internal. We have double blind experiments because, frankly, human beings cannot be trusted to be objective, so we attempt to constrain ourselves as much as possible, building into our systems filters to bias.

    I think this is the core of the argument here.

  37. Report this comment

    MD Hays said:

    From the original editorial, what emerges is a need for two review stages. The first stage should use the double blind review dealing with only the pure science. The second stage is initiated immediately following. At this stage, the editor reveals the author group to the reviewer(s) and invites him/her to offer a sincere but brief discussion about what they may change about their science review considering their perceptions of the author group. More work, perhaps…better overall review, likely.

  38. Report this comment

    B. Kamenov said:

    Some comments on Maxine’s post (made on February 8, 2008 03:53 PM):

    We are offered new arguments on the double-blind debate. But do they make any more compelling a case for single-blind review being the best system? Well, all the points made are quite valid and important, but none is directly related to the topic of the blog. Points 1 and 3 (authors suggesting potential referees and reviewers seeing each other’s comments) can work exactly as well in a double-blind setting as in a single-blind one. Point 2 concerns the fact that almost all authors tend to cite their own previous work. True, but they almost always cite work from multiple other labs too, so if self-references are not explicit, reviewers will still be left guessing the identity of the authors. Point 4 is a straw man-type of argument. None of the commentators on this blog has so far even alluded to scrapping peer review in favour of preprint servers, and certainly no one has suggested a triple-blind refereeing system. In fact “triple blind peer review” is an oxymoron, because how could someone be a “peer” if he/she is selected blindly?

    I am fully convinced that peer review improves the quality of submitted papers in the vast majority of cases. However, in the extremely competitive world of science improving your papers is not sufficient for survival. The papers also need to be published, preferably in top tier journals. And in making publication-related recommendations, referees, who are also in for the race, will unavoidably have biases. Biases don’t have to be negative and are usually expressed in much more subtle ways than “a vague endorsement or a vague criticism on a submission”, which of course can be easily dismissed by a competent editor. There’s hardly a paper that cannot be dismissed with specific technical arguments, if a reviewer really wished to do so. Such biases are not “base motives”, but merely an inextricable part of human nature. Denying that is denying the very fundament of human psychology. It is time to simply face the fact that bias exists in every situation where humans are involved. The issue at hand is which system will minimize it more efficiently: single-blind or double-blind refereeing.

  39. Report this comment

    Adam said:

    I think the idea of a double blind review is a very good one. The arguments against it seem weak. Some good points are raised about the issue of reviewers slating work with impunity and the potential for retaliation against junior reviewers if their identities are know.

    I therefore propose: Double Blind, Double Open review.

    No one knows who anyone else is until publication – ones comments are freely accessible but your identity is not. Upon successful publication the reviewers identity and comments are made public. Allowing at least partial credit/blame for a reviewers opinions – given that publication was successful there is little reason to retaliate against a junior reviewer – who can indeed be rewarded for a well thought out review of a paper. Slavish ‘yes men’ will receive the appropriate amount of credit too… As will reviewers who are too negative (as they will never make it into print). Over time everyone involved in the review process will benefit – especially if they engage fully and review many manuscripts.

    It seems to be a superior system to Either Single or Double Blind review as everyone can make their contribution during manuscript submission without fear or bias and the reviewers receive some credit/blame for their efforts. Of course a good guess as to an authors identity can change the way you review but then if an article is successfully published everyone gets to see what you said.

  40. Report this comment

    David Berrigan said:

    Dear Nature, You accept that non double blinded review leads to bias against female associated names and yet you are sticking to your policy – shame on you!

    Davidina Berrigan

  41. Report this comment

    M. Rogers said:

    I’m very confused by Maxine’s comment #3. I have reviewed for many journals and I almost never knew (for sure) the identity of the other reviewers or saw their comments during the review process. I only saw the comments once a decision had been reached but still did not see the identity of the reviewers, although one can sometimes guess. So I don’t quite see how this is a “check and balance”. In fact, there is no accountability for reviewers, since the worst that can happen is that their review is discounted and that they are asked with less frequency.

    The real problem, in my view, is not that there is direct bias against junior scientists, but favoritism for senior scientists, especially those who are well known. Given journal space limitations,that does create a disadvantage for junior investigators. This favoritism is seen not only in papers but in grants. What fuels it? fear of retribution? fear of questioning “big shots”? fear that next time they’ll go to another journal? I don’t know but the bias is very real.

    So as I read the various comments, I find myself wondering the real motive for journals to resist minimizing bias or at least the appearance of it. It seems odd, especially when a crucial component of the peer-review process is credibility and from reading the comments from the many courageous junior scientists, is clear that credibility is lacking.

  42. Report this comment

    Sebastian Fugmann said:

    To Maxine:

    I’m very well aware of the fact that authors suggest/exclude reviewers. While this is a good method to work against part of the perceived bias, I think it would work equally well in the double-blind context. Unfortunately (for science in general), suggestions these days seem to be mostly made on the basis “who might give me a favorable review” and not “who might provides the most objective critique”. And, yes, I’m guilty of doing so myself. Based on my own anecdotal evidence (like questions from colleagues whether I got their paper to review as they suggested me as a reviewer) and causal conversations with editors at meetings (indicating they prefer to “use” the same small group of experts as reviewers on some topics) makes me interested in the actual number for “often” ( in “authors to suggest peer-reviewers (these suggestions are often followed”.

    I agree that most papers reference earlier work by the same authors. BUT, I think I could easily write my manuscripts to look as if they were coming from my postdoc advisor (a well-established and extremely successful senior PI) building in part on previous data coming from his lab … just an example how even seemingly obvious guesses on author identity in the “double-blind” process could be wrong (and may turn out to be surprisingly beneficial for junior scientists).

    I don’t think there is a global “conspiracy” but being in a field that is quite crowded and having heard comments like “you have to pay your dues first”, “you still have plenty of time to get your Nature paper”, “get in line, other people have worked for 10years on this question” make me wonder whether a second layer of thought comes into play when papers get reviewed in the single-blind system. Unfortunately all “evidence” to this extend is anecdotal.

    I agree that “disappointment” should not be confused with “bias”. But as a bias frequently leads to disappointment of the individuals suffering from it, it is one (but clearly not the only) marker for bias.

  43. Report this comment

    Maxine said:

    To M. Rogers:

    Nature and the Nature journals always send the peer-reviewers each others’ comments when a manuscript is revised and resubmitted; we also send all comments to each at the end of the process (accept or reject) to inform them of the decision and so they can see how the other referees evaluated the manuscript. We get positive feedback from reviewers about this practice. This has long been our practice, as is explained in our web page on the peer-review process.

    ————–

    There are some good points being made in this thread, which provide much food for thought. It is evident that some writers in this thread sincerely believe there is bias in the single-blind system; some feel that peer-reviewers are by definition biased and that the journal editors may not do a very good job at ensuring a manuscript is fairly assessed, so that more protection is needed for authors.

    How objective are we when we consider the significance of work we ourselves have done? Do people know of cases of bias that don’t concern their own work? It would be helpful to know of details (which can be sent in confidence by email).

    Nature used more than 5,000 different peer-reviewers last year, by the way.

  44. Report this comment

    B. Kamenov said:

    I have the impression that there’s a potential misunderstanding here, which should be prevented. It seems that that the journal editors involved in this thread feel that they are being personally blamed for the deficiencies of the current, single-blind peer review system (e.g. “some feel that […] the journal editors may not do a very good job at ensuring a manuscript is fairly assessed” from Maxine’s post). But I haven’t actually seen here comments that blame the editors for the bias-related faults of peer review. Indeed, in my understanding, there’s no real accusation (such as “conspiracy” or “base motives”) against referees either; they are simply seen as human beings, with the inevitable biases in judgment that come with that. I am not worried to admit that, although I earnestly hold objectivity as a most essential element of the scientific process, I personally could probably not avoid having subtle biases as a reviewer. And although not all reviewer biases would be based on the identity of the authors, some of them would, and they would be minimized if reviewers do not know who the authors are.

    Let me state for the record my personal conviction that the editors at Nature and many other journals are doing a great job at the tremendously difficult task of evaluating cutting edge scientific research. However, even the best editor is limited by the system within which he/she operates, and should from time to time consider seriously whether some aspects of the system can be improved. This is the one issue that I, and most of the people writing here, do take with editors – their refusal to adopt, or even test, double-blind review, which will inevitably reduce at least some of the biases existing in the current system. Some commentators may have used strong statements, but they are probably frustrated by the fact that the existence of biases and the potential for their partial avoidance using a double-blind procedure is so steadfastly resisted by editorial offices across the board. But would an editor be more receptive in the analogous situation where an author insisted that double-blind studies have no advantage over single-blind ones, because a couple of old studies found no difference in specific cases?

  45. Report this comment

    Bob O'H said:

    I’ve just been playing around with a simple model of the effects of double-blinding, and it suggests that it just shifts the bias so that the very famous and the obscure benefit the most. I’ve no idea how important it is in reality, but at least it suggests that the different ideas need to be examined carefully.

    Bob

  46. Report this comment

    Maxine said:

    Thanks, B. Kamenov, for your clarification.

    A question for you (or any science researcher reading this thread), is how would you devise an objective test of double-blind versus single-blind peer-review for a journal? What would be the success criteria of such a test?

    My point about the editors doing a good job (or not) is simply that the editors choose the reviewers (one aspect that they can do well or not well), and also that they judge the paper at its revision stages in the light of (usually) three independent peer-reviewers — from different groups, people who did not see those authors’ last submission, from different countries and so on. It therefore seems to me that, if the editor is doing her/his job properly in all these respects, there is no bias.

  47. Report this comment

    B. Kamenov said:

    To Bob O’H:

    It is clear that double-blind review will not eliminate all bias and may indeed skew some of it, as your model suggests. However, I think one can easily build an analogous model for double blind procedures applied to clinical or psychological studies (e.g. in a clinical trial blinding could also be unreliable, since doctors can often guess correctly which is the drug and which the placebo, based on clinical effects of the former). What puzzles me greatly is that double-blind procedures in reviewing scientific papers are so vehemently debated, whereas they have long been established as the undisputed gold standard in most other situations involving humans.

    In addition to being “arm-waving with numbers” the proposed model also paints a grotesque caricature of referees as people who, on receiving a manuscript, will have two preoccupations only: first, to make a guess at who the authors are, and second, to write a review that is based exclusively on their guess of the author identity. This may be the idealist in me speaking, but I believe that not only are reviewers not like this in the first place, but that widespread adoption of double blind refereeing will eventually diminish people’s tendencies to guess who the authors are and to act on their guesses.

    To Maxine:

    I’m sure we all agree that instead of plucking them “out of the air”, models should be built using real-world data. And that brings your question to the front: “how would you devise an objective test of double-blind versus single-blind peer-review for a journal? What would be the success criteria of such a test?” (thanks for putting the solution of that “trifle” issue on my shoulders, Maxine 😉 ). Obviously, this is the central question here, and only the opinion of the best experts in this area of research should be considered. But, anyway, being a complete amateur in this field, here are my two cents.

    The way it was done in the few previous studies on double-blind vs single-blind review that I’m aware of, was to compare the editors’/authors’ assessment of the quality of reviews that were done blind, with the quality of those that weren’t. Quality assessments were either done subjectively, based on predefined criteria (McNutt et al, JAMA 1990, 263:1371 – found a difference; Justice et al, JAMA 1998, 280:240 – did not find a difference) or were calculated based on the successful detection of deliberately introduced weaknesses (no difference detected; Godlee et al, JAMA 1998, 280:237) or on the likelihood of recommending rejection (difference detected; Godlee et al, JAMA 1998, 280:237). There are admitted limitations of these studies, such as poor masking success (which could have been improved using better procedures) and the awareness of the reviewers that they are participating in an experiment. In addition, high quality review is not necessarily the same as unbiased review.

    Another way to go about it would be to send a group of papers to two sets of referees. The papers should be chosen according to the following criteria: 1) they should pass the “editorial filter”, i.e. be considered potentially suitable for publication; 2) they should be truly unknown to the reviewers. The latter can be achieved with high likelihood by asking the authors whether they have presented any of the data in talks, meetings, preprints or personal communication. One group of referees should then get the papers with the real names of the authors and another group with substitute author names, ideally without letting the reviewers know that they are involved in an experiment. Papers stemming from famous authors can then be presented as coming from little known (or fictitious) researchers in a minor institute. For better balance, papers coming from rather unknown authors could also be presented as being written by “big shots” in the respective field. Although ethical considerations could conceivably be raised by such an approach, this is perhaps the best way to determine whether there is a correlation between an author’s standing and the likelihood of receiving recommendations for acceptance.

    A third approach that I can imagine is to plot the type of graph that Bob O’H did (fame vs acceptance rate), but using actual data. The “fame” of the (leading) author can be determined by, say, multiplying the number of his/her publications by the impact factor of the journal in which they were published, considering only journals with impact factors above, say, 10. One can already plot this graph for papers that were published under single-blind review and the same could be done for double-blind review, if it were tested. If such a test were undertaken, as many of us would hope, I would avoid trying to determine masking success in parallel, because if reviewers are specifically required to guess the identity of the authors, this may send them on the wrong track of mind, tainting part of the rationale behind double-blind review. If the single-blind and double-blind lines on the plot are slanted differently, then one should act accordingly. And, to be honest, whichever way the difference goes, barring some startlingly egregious result, I would switch to double-blind review.

  48. Report this comment

    Martin Taylor said:

    In my long-past days as a reviewer, I always asked to be identified to the author, except when I thought that the author might be junior and would accept my word as authoritative. My reasoning was first that if the author was required to stand behind the content of the paper, I should be required to stand behind the content of the review, and second that the author probably knew more about the specific work presented than I did. In case of dispute, the author ought to be able to know the background from which my comments derived.

  49. Report this comment

    B. Kamenov said:

    One additional point that has been alluded to here, and expanded upon in a related blog is that editors reject a large fraction of manuscripts before any reviewer has had a chance to see them. So, if bias is inevitable, blinding only the reviewers may not solve the bulk of the issue. This point is very well taken, but it’s not too hard to think of a way to address it: editors should simply be considered a part of the “evaluation team” in a double-blind system and should take their decisions without knowing the names or affiliations of the authors. With electronic submission that should be easily achievable. Although editors at Nature and a subset of other journals are not directly involved in scientific competition, other editors are, and nearly the same arguments would apply for both groups of editors in a double-blind system, as have already been discussed here for reviewers.

  50. Report this comment

    Bob O'H said:

    B. Kamenov – I think the comparison of double blinding in clinical trials with double blinding in refereeing is misleading. The purpose of double-blinding in trials is to make sure that the object of study isn’t affected by knowledge of the treatment. But I don’t think a manuscript cares one way or the other.

    If there is a comparison it’s with the doctor, who might be able to guess if a patient has a treatment or a placebo. But this is also not a good analogy, because the only way he should be able to to this is through the outcome of the treatment. IOW the trial is designed to test if a treatment works. If it does not, then the doctor’s guess is random. if it does work, then the guess is skewed, but the effect will be to increase the effect size, i.e. to make any effects clearer. But if the aim of a trial is to find out if a treatment works, then any bias that is only affected by whether the treatment works is not a problem. It will affect the effect size, but surely only by a fraction of the “true” effects size.

    The other aspect that’s important is that blinding of a manuscript might not work because it will carry with it clues as to the identity of the authors. A red sugar pill doesn’t do this – most discernable differences can probably be masked, but it is almost impossible to do this with many manuscripts (remove all the references? Tell the author that they cannot mention their study site or the organism they are working on?).

    On my “grotesque caricature”, yes obviously it is. That’s what models are – one isolate the phenomena one is interested in, and only studies that. How much of a caricature it is is an empirical question, hence my plea for data.

    Maxine – B. Kamenov has described the sorts of approaches that could be used to measure the effects of blinding. The Justice et al. and Cho et al. papers cited in the editorial describe data that would be suitable – “fame” could be measured along the lines that B. Kamenov suggests (I think I would count the number of citations of papers, and perhaps take the author with the most, or a sum or mean over the authors, but these are details).

    Bob

  51. Report this comment

    W. Hornfeck said:

    In my opinion peer-review is always biased as long as it is not possible to let computers do it. A review will always depend on several very subjective factors (e.g. the general attitude of the reviewer to the review process, the time he is able to spend, his own interest in the topic of the article, even his mood, …), even if the reviewer tries to circumvent that. With the current policy the disadvantage is clearly on the side of the authors, because in what would be suspected to be a case of an ill-founded rejection of the manuscript there is nearly no way to straighten out the problem. On the other hand there are clearly syndicates of people advancing each others articles and it is also true that people appearing more famous than others are able to publish their research in a less reviewed way (e.g. as the cases of Woo Suk Hwang or Jan Hendrik Schön gave proof of). I would argue that peer-review in total is overestimated, as it is not really proven that it has the capability to figure out, e.g., if someones research is based on true results or not. If the results seem important to someone, the research will always tried to be reproduced, if it is judged unimportant, who cares? I think it would be advantageous for the scientific community in general to cultivate open-mindedness and transparency to the highest possible degree. Secretiveness, unless it affects your bank account, rarely does anything good by itself, it rather is a hiding-place for indolent and careless reviewers which are not really forced to stand behind their judgement.

  52. Report this comment

    Maxine said:

    Refs 2 and 3 in the editorial, and referred to by B. K. and Bob O’H, are part of a special issue of JAMA http://jama.ama-assn.org/content/vol280/issue3/index.dtl, and show that blinding peer reviewers did not have a measurable effect. (As stated in the Editorial reproduced in this post.) There are other articles in that JAMA special issue showing lack of other effects, for example blinding reviewers did not affect numbers of errors identified by the reviewers.

    Elsewhere in this comment thread, the phrasing in our editorial, paragraph beginning “The one bright light…”, has been misunderstood. This sentence is highlighting one reported advantage in the context of other demonstrated non-advantages. And, according to Bob O’H., this “one bright light” (the TREE study cited in the editorial as ref 4) may require modification of its conclusions.

  53. Report this comment

    B. Kamenov said:

    Bob,

    You write that “The purpose of double-blinding in [clinical] trials is to make sure that the object of study isn’t affected by knowledge of the treatment.” Well, actually this would be the purpose of single-blinding. The reason double-blinding is always preferred is that the doctor can also influence the outcome, usually by very subtle tendencies to treat the drug-receiving patients differently from the rest (i.e. in the same way that a reviewer could affect the balance between the frequency with which, say, famous and non-famous authors, get published). There’s also another reason why the analogy between blinding a reviewer in manuscript evaluation and blinding a doctor in a clinical trial is better than you suggest. The final effect of the drug on the condition for which the trial is designed is by far not the only clue that a doctor can get. That would be like saying that the appearance of the accepted paper in print is the only way in which the paper’s reviewers could find out the names of its authors. A large percentage of drugs have pronounced, readily detectable side effects, which usually appear before the final outcome of the intervention can be assessed. Such side effects of a drug are an excellent analogy to the clues a manuscript may contain to the identity of its authors. I would in fact think that enforcement of neutral writing practices may make manuscript authors harder to detect than drug vs placebo.

  54. Report this comment

    B. Kamenov said:

    Maxine,

    The issue of JAMA you mention covers different aspects of the peer review process and only a few papers are related to the question of what the effect of masking author identity is. The common thing between these papers is that they compare mean values for review quality between blinded and non-blinded groups. However, if non-blinded reviewers were giving somewhat higher quality reviews to one group of authors (let’s arbitrarily call them “famous authors”) and somewhat lower quality reviews to another group of authors (let’s call them for contrast “not so famous authors”), those differences could cancel each other out and show no distinction between blind and non-blind review in this kind of comparison, although bias would in fact have been pervasive. That bias can indeed affect reviewers significantly is demonstrated by a further study in the same issue of JAMA (Link, JAMA 1998, 280:246), which showed that US reviewers recommended acceptance of papers by US authors more often than did non-US reviewers.

    Another point is that all of the JAMA studies were limited to medical journals, which may represent a special case. That’s a further reason why they should not be used as a justification for considering the issue of double-blinding definitively solved for all kinds of journals, including multidisciplinary ones like Nature.

  55. Report this comment

    Maxine said:

    Dear B.Kamenov

    Geographical bias is not an issue for Nature, an international journal (as stated on the cover), we use referees from everywhere, and go to efforts in this direction (our editors are from a very wide range of countries also). If possible, when there are appropriate referees, we use referees from different countries for one manuscript. Also, compared with JAMA I suspect, Nature probably has a wider geographical range of authors — both over the whole journal and per individual paper.

    I understand your perspective and you make some very good points, I think. However, you seem to be suggesting focusing more and more journal resources on devising ever more complicated tests of a system that we (for the reasons described on our peer review pages) think is the best system, given the practicalities of running a journal, resource issues, time issues, what peer-reviewers and editors (all overworked) can practically do, and so on. As well as this, all evidence (as opposed to opinions, which differ) are that the system is not biased, apart from that geographical bias paper you mention.

    I also think that some of your and other’s suggestions about judging an authors’ status are highly subjective as well as impractical to impliment in terms of a trial — and in any event, as mentioned, the average number of authors on a Nature paper is 5, with a range of “statuses”. What would you do about a consortium of 100 plus authors for example? (Rhetorical question.)

    There are several possible systems of peer-review and they all have their proponents. We ran a series of 22 articles eighteen months ago going into all of this, you can read them on this blog.

    One big problem for journals, perhaps the biggest in the sphere of peer-review, is finding people who are prepared to even do this work at the level and on the timescale needed. Frankly, identifying good reviewers is more of a priority for a “working journal” than designing fiendishly complex trials that whatever they show will not receive universal agreement — as demonstrated by the JAMA studies. The level of author keenness in the double blind system has also been tested in practice by the PLOS journal mentioned in the editorial. It was low.

    That having been said, we are going to be discussing all feedback (not only via this comment thread) very soon and will consider our peer-review process in the light of all of it.

  56. Report this comment

    B. Kamenov said:

    Dear Maxine,

    Although I am not an editor myself, I have a long-standing interest in and respect for the editorial profession. I think that I have a fair grasp of the complexity of the tasks an editor has to perform in order to keep a journal running and publishing top notch science. I believe that I also understand some of the pressures that may be making you reluctant to challenge the single-blind peer review system. However, practicing scientists certainly have their own set of pressures and are in turn entitled to put pressure on journals when they feel they are not being served as well, or as fairly, as they could be.

    Out of respect for the importance of your main work, and mine, I intend this to be my last post on this blog. I feel that both of us have invested enough of our time addressing each other’s points, and I suspect that this is a feeling you may share. There are however, a couple of points in your entry that I would like to comment on.

    You seem to question the very existence of bias in the people involved in the peer review system (“all evidence (as opposed to opinions, which differ) are that the system is not biased, apart from that geographical bias paper you mention.”). I certainly cannot claim to be familiar with “all evidence”, but in JAMA alone there is at least one other paper that also supports the notion of the existence of bias in the current system: McNutt et al. (JAMA 1990, 263:1371), which reports a better quality of reviews when the reviewers are blinded. Furthermore, for reasons I have detailed in previous posts, the way review quality was compared in the rest of the studies cited on this blog, cannot exclude the presence of bias, even in the cases where no statistically significant improvement in double-blind reviews was detected. Puzzlingly, bias in scientists is well established (e.g. in the form of “publication bias”, also discussed in the 280(3) issue of JAMA); however, these same scientists are assumed to mysteriously lose all bias once they put on a reviewer’s hat.

    The low author keenness for anonymity seen at PLoS can be easily understood without assuming that the authors favour the single-blind approach. If I were offered a choice between masking my identity or not, I would go for the latter, because I would fear that if I opted to remain anonymous while others didn’t, I would be assumed to have something to hide, such as a modest publication record or affiliation. Because of such caveats, if the double-blind system is to be tested seriously, it has to be compulsory.

    You make it clear that the editors at Nature consider the single-blind system to be the best. The reasons for that are supposedly “described on our peer review pages”. I have gone carefully through the peer review pages that you indicated, and although I share your enthusiasm about the positive effects of peer review in general, I have not seen a single reason being cited, which addresses the question of why single-blind peer review should be better than any other possible form of peer review. In your comment you do mention some limitations that presumably should restrict the practical applicability of other forms of peer review (“the practicalities of running a journal, resource issues, time issues, what peer-reviewers and editors (all overworked) can practically do, and so on.”) However, I do not understand why, in an environment using automatically generated manuscript files by an electronic submission system, evaluating a manuscript that lacks the authors’ names should put any additional strain on a journal’s resources, on the time of the people involved, on “what peer-reviewers and editors (all overworked) can practically do” or on any other “practicalities of running a journal” that I can conceive of. As for the previously discussed studies that have compared single- vs double-blind review, none of them has found single-blind review to be superior and one has found it to be inferior to double-blind review. So having gone through all the arguments raised so far, I am still at a loss as to why single-blind review is so resolutely believed to be the best system.

    Regarding the possible ways of testing reviewer bias and the effects of double-blind review that I sketched in a previous post, they were only put forward in response to a question directed at me personally and were never intended to push your journal towards “focusing more and more journal resources on devising ever more complicated tests”. As mentioned above, I am aware that you have enough to do as it is. The one point that I did want to make by outlining these possible tests was that bias and the potential advantages of a double-blind system are not abstract, untestable concepts, which should be left out of the discussion. Bias and the effects of double-blinding can be tested empirically and objectively.

    However, for practical purposes I am not suggesting that you spend time and resources on trying to determine whether double-blind review could be beneficial, while running in parallel business as usual. I am suggesting something that is a lot less demanding – to simply adopt a double-blind system for, say, a year, and see what happens. That should give you enough data to determine with much greater confidence than previously possible what double-blind review can and cannot do for the scientific community. There is hardly any risk or any major effort involved, but the potential benefits, even if small, make it worth trying.

  57. Report this comment

    Leonardo de Oliveira Martins said:

    What an excellent thread! I’m impressed as how objectively we can detect the contribution of each one of the authors, with misunderstandings being corrected and clarified – thanks Maxine! I cannot imagine such an informative post in an anonymous environment.

    BTW, I couldn’t agree more with Matt Chew (February 6, 2008 10:20 PM).

  58. Report this comment

    Shi V. Liu said:

    Double-blinded or totally open up?

    ————————

    Single-blinded peer review has allowed some faked discoveries to be published as landmark papers. Could double-blinded peer review give some opportunities for those discovery-fakers a better chance to slink in again?

    Double-blind review will not work. The real solution is OPEN REVIEW.

    Nature once started a test on open review for some paper. Why should it discontinue it? I think if Nature has the gut to make every submission to be openly reviewed before and after publication, then authors have to live with it if they still wish to get the high impact factor from Nature.

    —————

    Shi V. Liu (SVL@logibio.com)

  59. Report this comment

    AB said:

    As a female author in a developing country I agree with the need to make peer review anonymous. If 50% of the time the reviewers can guess the authors then so be it but the other 50% may be benefiting from this system. I am a PhD student in a field I am not known in and worked with a senior person that I did have much respect for scientifically unfortunately his name was last author on my papers and therefore those in the same field that have a similar opinion to me would have automatically formed an opinion about the paper although the work was from my own ideas. In this way complete anonymity would have made it easier to be taken seriously.

  60. Report this comment

    RM said:

    One thing I would caution about in this discussion of peer review is not to forget the ultimate benefactors of the review process – the readers.

    Dinesh proposed that "There should be a “complete double blind” review … authors should be told a priori not to use phrases like “our previous work (ref) shows that”, " in continuation to our previous work", etc."

    I believe that pursuing double-blind reviewing with such zeal as to prohibit such contextualizing statements would be a severe disservice to readers of the paper. All too often authors treat previous work from their lab as if it was independent confirmation. Readers are well served by statements notifying them of previous related work, especially if this work is part of a body of knowledge coming from a single lab. Regardless of the value or valuelessness of knowing the authors’ previous body of work to the authors or reviewers, it is of great value to the readers of the paper to know how this work fits in with an existing line of research – especially if the authors have any biases of perspective which may carry through their research.

    I do not currently have an opinion on which review system (single-blind/double-blind/open) should be implemented by Nature or by other journals, but in all this discussion the true purpose of research papers must be kept in mind. Research papers exist not to pad the authors C.V., but to communicate valuable scientific results to readers and to the general public. The review process should be crafted to maximize the value to the reader, not to maximize the ego massage to the authors/reviewers/editors.

  61. Report this comment

    disdaniel said:

    “identifying authors stimulates referees to ask appropriate questions (for example, differentiating between a muddy technical explanation and poor experimental technique). Knowing author identities also makes it easier to compare the new manuscript with the authors’ previously published work, to ensure that a true advance is being reported. And knowing rather than guessing the identities of authors encourages reviewers to raise potential conflicts of interest to the editors.”

    In other words single-blind review promotes lazy reviewers to bias their review.

    It seems to me that you could send papers for double-blind review to eliminate bias…if this is a concern, Nature editors apparently are mighty comfortable with their biases. After review but before acceptance Nature could then ask the reviewer if they had any special (conflict of interest) concerns if the paper was authored by individuals a), b) or c). Nature could get both a less biased review, but still catch serious conflict of interest/newness concerns.

    This entails an extra step, but if bias is a concern—and the concern for editors should be that good science is not reported due to bias in the review process—then double blind plus targeted follow-up should help.

  62. Report this comment

    Sebastian Fugmann said:

    I agree with B. Kamenow that a well-controlled study to see whether double-blind or single-blind might take a lot time and effort (and potentially interfere with the current review/publishing system at Nature).

    A short term suggestion to gather more “preliminary” information (specifically what authors/reviewers think about double-blind specifically in the case of Nature):

    Given all the comments here on this blog I was wondering what the remaining 99.999% of the scientist who didn’t see/comment on the Nature Editorial and or this blog (or who are worried about publicly posting “non-conforming” comments – some colleagues of mine were amazed that I was crazy enough to post a viewpoint that is not in line with Nature, and thought that using my real name might have a severe impact on my scientific career). I think it would be feasible to send out all abstracts/manuscripts to (potential) reviewers without disclosing the name of the authors, and simply count how many reviewers actually realize that this information is missing. A simple questionnaire of why their review would benefit from knowing the identity could give a hint about what the actual reviewers think about the double-blind option (and the author names would be provided upon receipt of the responses). Similarly a short questionnaire to the authors asking them whether it would be OK with them to send their manuscript out without identifying them could provide more information about what the authors think about it. I think this would be a really simple way to get some input from those who would be affected without actually changing anything about the existing system and interfering with the review (and who would refuse to fill out a little questionnaire if Nature would make it a requirement – I guess authors and reviewers would be more than happy to comply).

    If there is an overall positive response to a double-blind system then I think it might be worth to risk implementing it (with the option of going back to the single-blind system in case there are severe problems that nobody had envisioned).

  63. Report this comment

    Maxine said:

    To Disdaniel: Nature journals do ensure independence(including lack of competing interests) of potential peer-reviewers before they are even sent the manuscript to review. You might like to read our peer-review guidelines.

    Sebastian Fugmann— thanks for your comment and your suggestions. I can assure you that no comment on this blog, or any other online comment by a user of nature.com, will have the slightest effect on how Nature or other Nature journal editors consider that person’s manuscript submissions— I am not sure if this is your precise concern about frankness, but you can rest assured on our account. As well as this being our editorial policy, the registration systems are separate for online comments and manuscript submission to a Nature or NPG journal 😉

    Nature editors have to get used to receiving many, many strongly worded and even unpleasant letters during their time at the journal. None of these make any difference to our consideration of future submissions by the authors of those letters. (It also makes no difference if people send us nice letters, though they are more pleasant to read!)

  64. Report this comment

    Aristotelis Tsirigos said:

    No matter how someone “reads” the surveys on this issue, the single most unmistakable fact is the growing discontent in the scientific world – especially among young scientists – with the current review process, both in terms of scientific article reviews and grant proposal reviews. In my opinion, anonymity, when it is one-sided, it naturally encourages unprofessional behavior from the reviewers’ side, i.e. reckless, biased, or generally low-quality reviews.

    However, I believe we are facing a much larger issue here: the debate is intentionally framed as a battle between authors and reviewers, whereas the fact of the matter is that the editors are the ones who have the power to make the final decisions, and should therefore bear the full responsibility. Instead, in several prestigious journals, they are also anonymous to the general public, i.e. nobody is allowed to know who edited a published article. But, let’s pause for a second and think how easy it should be for an editor (who should be an expert in his or her assigned field) to distinguish low-quality from high-quality reviews, or biased from unbiased ones! Then, what should an editor do, when a review does not meet the necessary criteria? He or she should simply reject the review, and should even go further and never ask that particular reviewer again for a review. This is how the editors can safely improve the quality of reviews. But, I am asking how often does this really happen, given that biased, low-quality reviews are encountered so often? I think we all know the answer and it is rather depressing.

    I really hope the editors have the courage, the will and the capacity to truly assume their responsibilities and work towards more fairness and more transparency in order to salvage a corrupt system that none of us deserves. Otherwise, our best and brightest students will choose not to become scientists, and the rest of us will end up publishing on our personal websites and effectively render the publishing business obsolete.

    Maxine notes: Nature journal editors are listed on the online “about the journal” pages of the journal concerned, together with their areas of expertise.

  65. Report this comment

    MolBio Grad Student said:

    I disagree rather strongly with the conclusions of this editorial. First, and most importantly, this study acknowledges that gender bias exists in the review process. It would seem a moral imperative to counteract this pernicious bias, whether or not the quality of reviewing itself improves. Newcomer bias is also perceived to be a problem (at least among newcomers), and since newcomers are in the minority among publishing authors, their views may be muted in survey results.

    The fact that reviewers can accurately identify an author 40% of the time mitigates these issues, but only by 40%. Even less, in fact, given the doubt that reviewers may have regarding their guesses. And, importantly, guessing an author’s identity is not possible when reviewing a newcomer, so this type of bias would be most effectively counteracted.

    As to the belief that single-blinding helps in “differentiating between a muddy technical explanation and poor experimental technique” — I see this as an argument for double-blinding, in fact. We should hold the same standards for experimental technique as we do for communication of these techniques. In my opinion, methods for well-known authors are often less clear (as they’ve been “given a pass”) — this is a disservice to the scientific community.

    I also find curious the statement that “double-blind peer review is at odds with another ‘force for good’ in the academic world: the open sharing of information.” If double-blinding goes against the “information wants to be free” meme, then why not identify reviewers as well? Also, the cultures of the physical sciences and life sciences are quite different, making it difficult to draw conclusions from their respective publishing habits.

    I think you should be guided by the number that takes the most factors into account — the amount of confidence authors have in the review process. You can try to explain away the fact that more authors have confidence in double-blind reviewing, but I remain wholly unconvinced.

  66. Report this comment

    E. Martinelli said:

    “I am still at a loss as to why single-blind review is so resolutely believed to be the best system.” This sentence, posted here by Kamenov, says it all.

    The only advantage Editors and Reviewers have from reading the author’s names is to be influenced by them and, accordingly, pay more or less attention to the manuscript. I wonder if there is anybody who can truly say he does not consider the name of the last author before starting reviewing the work…

    “Otherwise our best and brightest students will choose not to become scientists…” says Aristotelis Tsirigos. Well, I am a young scientist and I account this lack of fairness in the review process among the reasons why I am trying to change job.

  67. Report this comment

    José J. Lunazzi said:

    Well, but what about rejecting a manuscript without reviewing it?

    The discussion holds for this case too. Would in this case be necessary to send an article through a pseudonymous?

    I agree that third-world countrie’s scientists could be biased and would like Nature to make an article on it, with a corresponding blog. On this point, the language style would still be a way to detect them, even in double anonymous processes.

  68. Report this comment

    Sharad Kumar said:

    I am completely in favor of double blind reviewing and disagree strongly with the editorial. Anything that reduces reviewer bias can only be an improvement to the current peer review system.

  69. Report this comment

    Elina said:

    There is at least one study about the benefits of double-blind review:

    Budden et al. 2008. “Double-blind review favours increased representation of female authors”. Trends in Ecology and Evolution. 23:4-6.

    Maxine notes: This study is cited in the Editorial that forms the post here. See also discussion in the comments about the modelling of that paper – the conclusion of the paper may need to be modified.

  70. Report this comment

    Shi V. Liu said:

    Double-blind peer review wouldn’t work. Please try double-open peer review which means open review before and after the publishing of a manuscript. This can be done so easily now and there should be no “space” limitation in publishing all the reviews.

    Shi V. Liu

  71. Report this comment

    Yassen Pekounov said:

    Dear readers and Nature Staff,

    It is obvious that double blinded review will be beneficial! It is objective truth and this is our main responsibility to accept the objective truth as researchers, right?

    The single blinded review is excellent ground for favoritism, prejudice, etc. As always when someone act anonymously thus avoiding responsibility for his actions.

    Maybe the best practice available today is the review process of the open access journal BMC Musculoskeletal Disorders. Briefly, all reviewers are listed with their names along with all reviews and all revisions of the paper.

    This makes great science. And is fair.

    Regards,

    Ya. Pekounov

    Researcher

  72. Report this comment

    Phillip Gienapp said:

    I am strongly in favour of double-blind review. Even if the reviewers have a fair chance of correctly guessing authorship they can not be 100% sure, as I know from own experience: I correctly guessed the ‘lab’ in which the work was done but assumed the manuscript was written by a grad-student because it was somewhat sloppily written. Later it turned out that it was actually solely authored by the group-leader.

    Another issue are unjustified comments from native speakers on language or style of manuscripts by non-native speakers (inferred by name and affiliation). I had one manuscript rejected partly because it was “so poorly written and difficult to follow…” When I challenged that decision and was allowed to re-submit the manuscript was eventually accepted without any(!) language related comment. I think this shows that there’s quite a bit of variation in ‘taste’ among native speakers and that they may be a bit more careful with such comments about ‘poor language’ if they can’t be sure the manuscript is actually written by a non-native speaker.

    Finally, I think the strongest argument in favour of double-blind review is that it may have the huge benefit of a fairer manuscript assessment but that it has no major disadvantages. So, why not do it anyway?

  73. Report this comment

    Nina Papavasiliou said:

    Last week, the NIH released the draft recommendations from a panel convened to assess peer review (which for the purposes of the NIH is grant review). The document can be downloaded as a pdf file from http://enhancing-peer-review.nih.gov/).

    One of the recommendations (which are presently being discussed by the community prior to formal approval) is for anonymous (double blind) review (pg 44-45). I copy from the document:

    ————

    CHALLENGE 3B: Knowledge of the identity of an applicant/applicant’s institution might bias reviewers.

    Consultations with the extramural community yielded the comment that the practice of “blinding” applicants, if truly achievable, may reduce review bias. The Publishing Research Consortium released an international study8 on the perspective of peer review

    in scholarly journals (21). The majority of those surveyed (71 percent) had confidence in double-blind peer review, with 56 percent expressing a preference for this approach. However, several publications conclude that for journal articles, masking the identity of the authors had no impact on the quality of review (22-24). Moreover, there is a widespread view that the identities of the authors are difficult if not impossible to blind given a reviewer’s knowledge of their field. However, it has been documented that after a policy change by the journal Behavioral Ecology to blind author identity, a significant increase in the number of female, first-author papers was observed (25). The applicability of this observation to peer review is supported by observations reported earlier regarding the peer review of postdoctoral fellowship applications in Sweden (26).

    RECOMMENDED ACTION: Pilot anonymous review in the context of a two-level review system such as the editorial board model.

    The NIH should consider the value of anonymous review; however, piloting this concept necessitates two-tiered (e.g., editorial board style) review. The first, anonymous stage would assess scientific merit. The second, non-anonymous stage would take into account the investigator and his or her environment since these issues are critical to a project’s ultimate success.

    ————

    Notice that, since a paper submitted to a journal presents work already finished, only the anonymous stage would be required (ie no reason to guarantee against a project’s ultimate success since it is completed). If the administrative behemoth that is the NIH is contemplating this, journals like Nature really ought to give it a fair shot.

  74. Report this comment

    Jianfei Hu said:

    Currently, there are three major kind of peer review process: open, single blind and double blind.

    Open, the identity of reviewer and author are open for both, is the worst of the three. In one way, it is difficult for a reviewer to reject a manuscript if he will meet the author in later conference, especially when the author or corresponding author is an authority. Thus, it will lead to a high accepting rate, and decrease the quality of published paper. In the opposing way, it will reject some high-quality paper if the current author has rejected some papers of current reviewer, and exacerbate the relationship between scientists.

    Single blind, the identity of reviewer is secret while the identity of author is open, is used most frequently today, and has proved itself. The major disadvantage of it is that, when rejecting a manuscript, the reviewer does not need consider the incurred influence from the author. Thus, it is particularly unfair to young scientist (such as postdoc) and non-native-English-speaker scientists, since, in some cases, the judgment of the review is not based on the content of the manuscript but based on the identity of the author, which university, laboratory, or country the authors are from and how many paper or high-impact paper they have published, et al.

    Double blind, the identity of reviewer and author are secret for both, is very like the KEJU exam system in the history of China, which had worked more than 1,000 years and did a great help on the selection of eligible officials. The papers have to be transcribed by some employed workers before they are sent to the reviewer, to avoid that the reviewer can recognize the identity of the author from their handwriting. By this way, it can avoid the bias of identity, and poor and no-noble clerisy can also be selected as official as long as they can show their ability in their papers.

    I prefer the double blind since it can avoid the flaw of the open and the single blind.

    Jianfei Hu

    Researcher

  75. Report this comment

    Ana said:

    Double blind review would be beneficial in some fields. Field sciences and poor countries will not benefit because non-objective reviewers can still guess who the authors are or where the work was done.

    I would rather have open reviews where no one can hide and all comments on the paper (and authors answers) can be accessed online. No need to publish them with the paper, but they should be made public so no one can hide.

    Peer-review should bring scientists together in open discussion, not putting some scientists (anonymous reviewers) on a power position to even say things (I am quoting from a recent reviewer) like “How would I know, this is not my area” (?)

    No doubt, the system as it is needs further improvement.

  76. Report this comment

    Heather Etchevers said:

    I just read Bob O’Hara’s letter and felt inspired to respond to your call for opinions.

    First, in response to “The one bright light in favour of double-blind peer review is the measured reduction in bias against authors with female first names… This suggests that authors submitting papers to traditionally minded journals should include the given names of authors only on the final, published version.”

    As a female junior scientist in a field where seniority counts (and I don’t want to make accusations of sexism, but the numbers are out there), I like the idea of double-blind review very much, but have two problems with it.

    The first is that measures of hiding the first name or affiliation will not work if (a) any reviewer is acquainted with the laboratory, and a well-chosen expert is likely to be (b) only the people who stand to benefit from this measure use it. It must be an across-the-board rule.

    Bob’s idea of diverse models of submission only would work if the diversity is one model per journal. Clearly, from PLoS’ experience, a choice of different review systems backfires. (Interestingly, as PNAS and other journals can attest, single-journal diversity in models of paying/open access seems much more successful.)

    Second, I readily admit to bias when I see work from a laboratory whose past production has inspired either respect or skepticism. I will be less or more inclined to be critical. Overall, it will make my reviewing job easier or harder. Double-blind, if it could be perfectly applied, should get around this and make me work equally hard for all reviews. It might make editors’ jobs more difficult, though.

    One temporary solution is this: should a person choose to review for a single-blind journal such as Nature, their review will be made available as online supplementary content, and this, not the article itself, should be open for comments from the community. It would get the ball rolling for wider critical contributions. Then the editors could see how the reviews are received more widely, so as to choose (or not) that reviewer again; the reviewers are partly accountable without fear of direct reprisals, since if the article was awful, it wouldn’t probably have been published. One could hope.

  77. Report this comment

    Jose A. Stoute said:

    I have read with great interest the editorial entitled "Working Double-blind: Should there be author anonimity in peer review?

    I disagree with those who think author anonymity would not be advantageous. As some have said, I have seen very well-known scientists publish work with flaws that I know I would not be allowed to get away with if I was the leading or senior author. Again, as has been said before, even if the reviewers are able to “guess” the identity of the author, sufficient uncertainty will remain that will compel them to be as objective as possible.

    If openness is so good, why is it only good for the reviewers? Why don’t we have a totally open review system? Perhaps because there is the legitimate concern that authors will hold a grudge against reviewers. I personally think this concern is exaggerated as I think most scientists should be and are sufficiently mature to accept criticism. However, I think that journals may be drawn to protect reviewers due to a legal responsibility. An open system will make the reviewers more accountable for their comments and they will think twice about making unjustified demands or criticisms. I recently experienced the open review system as an author in a biomedical journal and I was pleased with it. The only draw back is that it took a while to find reviewers who were agreeable to the open system. I think this reluctance will be overcome with time and increasing familiarity.

    To summarize, either double -blind or open review systems is better than the current traditional system that emphasizes the protection of the reviewers identity at the expense of the author.

  78. Report this comment

    Maxine said:

    A correction to the editorial has been publshed here: http://www.nature.com/news/2008/080604/full/453711c.html and the post above amended today 5 June 2008.

    The Editorial ‘Working double-blind’ (Nature 451, 605–606; 2008) referred to a study1 that found more female first-author papers were published using a double-blind, rather than a single-blind, peer-review system. The data reported in ref. 1 have now been re-examined2. The conclusion of ref. 1, that Behavioral Ecology published more papers with female first authors after switching to a double-blind peer-review system, is not in dispute. However, ref. 2 reports that other similar ecology journals that have single-blind peer-review systems also increased in female first-author papers over the same time period. After re-examining the analyses, Nature has concluded that ref. 1 can no longer be said to offer compelling evidence of a role for gender bias in single-blind peer review. In addition, upon closer examination of the papers listed in PubMed on gender bias and peer review, we cannot find other strong studies that support this claim. Thus, we no longer stand by the statement in the fourth paragraph of the Editorial, that double-blind peer review reduces bias against authors with female first names.

    References

    Budden, A. E. et al . Trends Ecol. Evol. 23, 4–6 (2008).

    Webb, T. J. , O’Hara, B. & Freckleton, R. P. Trends Ecol. Evol. doi:10.1016/j.tree.2008.03.003 (2008).

  79. Report this comment

    S. J. Melles said:

    Why not also cite the follow up article by Budden et al. 2008 TREE as they state that “The increase in female first authors within Behavioural Ecology (BE) was 7.9% whereas the mean of the other journals was 3.7% +/- 2.1 SD. So, BE falls on the 95% confidence interval (0.6%–7.9%).” Perhaps the jury is still out because there is not enough data and a lack of incentive for journals to make a change and see what happens! I am unconvinced by the results presented in Webb et al. 2008 TREE. They suggest that there was no significant interaction between gender and review type (-0.15 +/- 0.102, P = 0.134), indicating that the increase in female authorship over time in BE is not exceptionally different from the changes in the other journals in the field). What is their sample size? Do they really have sufficient power to detect a significant interaction effect with these data?

    In my opinion as long as there is a chance that a few scientists (e.g., Prof. P. Kantorek, of Ryerson University in Toronto Canada, see his blog post on a Nature blog below) are apt to complete single-blind reviews from a gender-biased or racist perspective, then a FAIR, SCIENTIFIC and “OBJECTIVE” system should work to prevent this: A double-blind peer review system prevents this type of bias because even if the reviewer guesses authorship, they can never be 100% certain.

    http://www.nature.com/news/2008/080423/full/452918a.html

    “My experience as a physicist working with the occasional female colleagues leads me to a subjective impression that women really think differently. Female thinking seems to be more lateral then vertical. By that I mean, women in physics are generally harder working than male colleagues and are great co-workers in terms of encouragement, diligence, and backup support. They do not, however, contribute a great deal of original ideas and rigorous logical analysis to the research. Female judgment seems to more emotionally biased.”

    P. Kantorek, posted 23 Apr, 2008

  80. Report this comment

    S. J. Melles said:

    As a follow up to my last comment, perhaps the movement towards a double-blind system has to start from the bottom up?

    I call on the 71% of researchers that have confidence in double-blind peer review and the 56% that prefer it to other forms of review".

    Simply refuse to review any more articles from journals that do not maintain a double-blind review system. Why continue to volunteer time to a system that you don’t entirely believe in?

  81. Report this comment

    Stephen G. Warren said:

    I oppose a “blind” reviewing policy. The list of authors is an important part of the paper and should not be immune to criticism by the reviewers. In three recent reviews, I criticized the author-list as part of my review. In one of them I recommended adding Scientist X to the author-list because the paper relied heavily on unpublished work by Scientist X, and the authors accepted my suggestion. In the other two I pointed out that the author lists were too long; I cited the “no hitchhikers” policy and other editorials on the topic of authorship that have been published in Nature. [One of the submissions had 127 “authors”!]

  82. Report this comment

    forex investment said:

    I think a double blinded peer review offers a truly unbiased platform. This allows for the research to be evaluated purely on the basis of the presented findings rather than the reputation of the author. As such, novel findings from lesser known PIs may find more honest review without any pre-existing biases on part of the reviewers.

  83. Report this comment

    Andrew Jenkins said:

    I think there is a lot to be said for ‘reverse single blinding’ – with the reviewer identified, but the author anonymous. Not knowing how senior the author is, the reviewer is likely at least to be civil.

    However, I would like to suggest going one step further along this line – publishing the reviewers’ names in the paper. It is my impression that sloppy reviewing is more of a problem than bias and have often had cause to remark “how did that get past the reviewers?”. By recommending a paper for publication, the reviewers, among other things, give their approval that the manuscript satisfies basic criteria such as completeness, comprehensibility and traceability.

    Printing the reviewers’ names would be a way to thank reviewers for doing a necessary, but unpaid job, with the sting in the tail that if they let a poor paper get published, they take their share of the blame.

  84. Report this comment

    Nacho Areta said:

    Dear Scientists, I am 100% in favour of an open system, where both authors and reviewers are explicitly shown. On the other hand, as a junior south-american scientist, I am fully convinced of the handicap that this brings to my career, especially under the current practice of keeping anonymous reviewers. Yet, I have nothing to hide, and I think the best way to improve the peer-review system , is to avoid hiding ideas, opinions and judgements. To me, science quality depends, among many sociopolitical issues, both on conscious authors and reviewers.

    Many times, anonymous reviewers express unjustified and negative opinions using an authoritative (and apparently irrefutable) style. Since many of these opinions will end up in the article not getting published in the journal where it was submitted, this hidden reviewer will certainly not be known to have acted as a reviewer, although it did. Also, a second journal might insist and send the same article (or a modified version) to the same anonymous reviewer, which might (for any reason, either valid or not) reject the work again. This is a rampant case of pseudoreplication in the review process. Whether this should be encouraged or not, can be discussed. Knowing the name of previous reviewers can serve both, the author and editors, to guarantee scientific quality.

    Excellent reviewers tend to be lacking, and most of the time they will only destroy what they can while not providing helpful suggestions to improve what can be improved. Their anonymous nature makes this way of reviewing all the way easier, while having their names attached to their evaluation might make them more eager to work harder on their evaluations.

    Given the lack of specific space in most publications to write the names of the reviewers and their concept of the paper, I support adding statements in the section of Acknowledgements such as “This work was negatively reviewed by X.Xi and W.Wu, while Z.Zo and Y.Ya found it excellent” (or similar statements, with varying degrees of precision as desired or needed to keep true to the review process). This would provide invaluable data to understand how scientists appreciate scientists works. And, indeed, science is a collective construction.

    If science is one way of trying to reach some kind of true knowledge, I do not see how we are to reach such a high goal by hidding events that relate to the production of such a knowledge.

    No more hidden reviewers. No more hidden authors.

    Saludos a todos.

Comments are closed.