Action Potential

Confidential comments – your opinion

Discussion is heating up regarding a new proposal that could change the face of peer review in neuroscience. At the PubMed Plus leadership conference this past June, sponsored by the Society for Neuroscience, the creation of a Neuroscience Peer Review Consortium was proposed.


Here is a message from SfN president David Van Essen describing the vision for this new entity:

After an article is rejected by one journal and authors are ready to submit a revised manuscript to another journal, they will have the opportunity and the option to request that the reviews from the first journal be passed directly to the new journal (assuming that both journals are part of the consortium). In many cases, the second journal will be able to reach a decision faster and more efficiently, thereby benefiting authors as well as the overly stressed manuscript reviewing system.

This revolutionary proposal is now a reality, at least for a trial run from January to December 2008. It will clearly benefit just about everyone in the community, including authors, reviewers and editors. The idea came from journal editors who saw that many technically solid manuscripts were being rejected because of space limitations or because the articles were not appropriate for their journals. After rejection, a new submission leads to another time-consuming review process, often with new referees, swelling the workload of the community.

But to make this new system work, the member journals had to agree on certain protocols to ensure that the inner workings of the journals are roughly similar. That way, the transfer process can go smoothly. One of those criteria got me thinking about our current review process. That is the “Confidential Comments” box. In order to join the consortium, journals will have to forego confidential comments.

For those of you who have not done much reviewing, many journals provide the opportunity not only to include a review of the manuscript for the authors, but also to communicate any additional feelings or opinions directly to the editors with confidence that the authors will not see these comments.

Although it is already clearly stated in our reviewing guidelines that the confidential comments and authors’ comments should match, sometimes they do not, which can put the editors in a difficult position. Obviously, ethical concerns and other issues of this sort need to be dealt with in a more private manner (as the consortium rules permit), but these issues only arise rarely.

We have had long discussions (arguments) in our editorial meetings about the benefits and disadvantages of confidential comments, and now want to hear what the community thinks. What is your opinion as an author? As a reviewer? Why should we continue to invite confidential comments? Why should they be banned? How can we improve our use of confidential comments to help us make good decisions while maintaining transparency with authors about the reasons for our decisions?

Further discussion here.

Comments

  1. Kendal Broadie said:

    Re: “confidential comments” to the editors

    As many of the editors of Nature Neuroscience are aware, I have consistentially refused to place any information in the “confidential comments to the editors” review box. This is because I feel strongly that this means of communication is inherently suspect and open to abuse.

    In my opinion, every least communication between the editors and the reviewers should be available to the authors. Moreover, I feel that the authors must be given the right to respond to any criticism of their work. It is simply not fair to allow confidential comments to determine editorial decisions without this ability to respond.

    In my opinion, the review and editorial decision must be as transparent and open to inspection as possible. The “confidential comments” to the editors clearly undercuts this openess. I can conceive of no beneficial function for this mode of communication, and plenty of abuses. Therefore, I urge you to please dispose of this mode of secretive communication.

  2. Douglas Nitz said:

    I could probably work around the absence of a ‘confidential comments’ section, but I do occasionally find it useful. It is useful if you want to tell the editor that the paper is great, but would benefit greatly from a new analysis or new presentation style. In other words, if the author thinks you love the paper, they’re less likely to agree to work on changes.

    Thinking about it more, it seems as if confidential comments might be very useful in the event the referee knows about a conflict of interest, knows about conflicting data from the same lab that exists but is not in the manuscript, or knows of similar work in another lab. I’ve never had this experience, but I imagine it will happen sooner or later.

    Why does the consortium plan insist on the absence of confidential comments? Presumably, if the author determines whether or not old reviews are to be considered, he will make that decision based on whether the reviews were good or bad. I don’t see how he is helped or hurt by the exclusion of confidential comments.

  3. Elizabeth Grove said:

    Confidential comments to the editor are not in all journals always absolutely confidential. If there is a disparity between the review, fullsomely praising, and the comments to the editor, much more negative, editors will sometimes disclose the comments. For example, if the author writes in puzzled, saying she can address those issues, and what about this being ‘one of the most important papers of the year’; why is it summarily rejected? There is in other words transparency if you are determined to get it.

    As a reviewer, I find the comments to the editor box useful when a paper turns out to be not so great. I can write as positive a review as possible, stating the negatives clearly, but then advise the editors more bluntly that this paper is not, and most likely never will be, for their journal. I agree the review should not be essentially different from the ‘private’ comments. It’s just that the latter can be blunt in a way that would distress the ego of many scientists.

  4. Noah Gray said:

    I wanted to forward an anonymous posting adding to the debate:

    I find the ability to make these confidential comments very useful – basically they allow me to say things that you as editor (or I as reviewer) might not want relayed to the author – like “I really want to see this paper published” or “It’s most critical if the author does X, but Y is not so important”.

    If I put that in my review, then the author would not respond to my criticisms seriously. My mentor, who was also a journal editor, trained me not to put comments like that into reviews because she felt it made her job harder as editor. Clearly, I want to convey great enthusiasm or what my key points might be to you, so that you will appreciate them in making your editorial decision.

    However, I generally try not to say things that are negative in the confidential comments, as I agree that this is really not fair – things that directly challenge or criticize the paper should be in the anonymous response.

  5. Peter Hollenbeck said:

    I find reasons to agree with both Dr Broadie and Dr Nitz. Confidential comments to the editor have the potential for abuse in at least two ways: they could provide additional, perhaps quite subjective, criticism of an author’s work that is then not available for the author’s use in revising the manuscript; they could, under the guise of “letting the editor in on things,” undercut the work with unreasonable, opprobrious, or irrelevant information. The reason for a reviewer to use the latter would of course be to escape the authors’ ability to detect and object to unreasonable criticism. This is simply wrong.

    As a reviewer I find the confidential comments occasionally useful in order to let the editor in on my reasoning in more detail than makes sense in a review. I use this as often to point out the importance or originality of a particular piece of work to my field as I do to explain why I feel work is flawed, inappropriate for the journal, etc.

    If the balance of views in the neuroscience community is that confidential comments to editors have too high a potential for abuse, then I could easily do without them in my reviewing. But note too that if the principles of fairness and transparency that Dr Broadie suggests are to be rigorously observed, then editors should also consider the content of, for example, personal or phone conversations with reviewers about manuscripts, and whether those too should be eliminated or perhaps divulged to authors.

  6. Kamran Khodakhah said:

    While I have found the confidential comments useful on occasions, I think I would actually prefer their omission from the review processes. It would be great to have a “what you see is what you get” review. With such reviews the authors would be confident that if they were to effectively address the concerns raised they will be able to publish their manuscript, or alternatively that there is nothing that they can do to convince the reviewer that their work is worthy of publication in the intended journal.

  7. Christof Koch said:

    I always prefer candor and have found that any and all criticisms can be placed into the comments accessible to the authors. Thus, I can live without these ‘private comments to the editor’.

  8. Chris Miall said:

    I have very rarely used the confidential comments section for any journal. In fact I find it very irritating that some journal review websites actually require a separate entry in the “comments to the editor” section.

    Almost everything critical but fair can be phrased in a way that is acceptable to the authors, even if it is unwelcome.

    In contrast, almost anything unfair might be said in a confidential comment that the authors are unaware of.

  9. Clifford Saper said:

    The discussion on confidential comments parallels that of the original working group that came up with the Neuroscience Peer Review Consortium. In general, many of us as editors had noticed that the confidential comments box was often used inappropriately by reviewers who placed important formative information about the manuscript, that should have gone to the authors, in a segment of the review that the authors never saw. We felt it was important that the review process be transparent to both the authors and the next editor who received copies of those reviews, and that the only way to do this was to eliminate the confidential comments to the editors.

    At the same time, we felt that there were some legitimate uses for the confidential comments, as stated by some of the other writers. It seemed to us, as Dr. Koch notes, that almost all formative comments could be put into a form that would fit the narrative review. The few comments that we felt would not fit, but were essential, were those related to ethical issues (human subjects, animal use, or scientific misconduct). We specifically recommended that there be a box for this, but that the journals should NOT pass on reviews of a manuscript for which there were ethical concerns until those concerns were adequately resolved by the first journal.

    We also noted that the telephone and email to the editor still exist, so that if one is asked to review a paper for which one has concerns that have to be passed on confidentially (such as the reviewer may have had access to some information along the way that would reveal the reviewer’s identity if mentioned), then this can still be done. However, as in the past, this would remain outside the formal review process, and could be dealt with in confidence by the first editor as required.

  10. Matteo Carandini said:

    I am ambivalent about this issue: confidential comments can sometimes be useful. Overall, however, I think that ditching them would make us write and read more informative reviews.

  11. Dario Ringach said:

    I would prefer to do without the ‘confidential comments’. In my opinion, ethical issues are pretty much the only case that would warrant such a private line of communication.

    If there is anything in a manuscript that can be modified to make it publishable, why not communicate it directly to the author as clearly as possible? If, apart from technical issues, the reviewer feels the study does not (and will never) reach the significance required for publication in a particular journal, why not say so directly within the body of the review?

    It is not unusual in top journals to have a paper rejected with reviews that, at first sight, appear to be very good. One is left wondering if more negative comments were transmitted confidentially to the Editor — criticism that, in principle, could be extremely useful to the authors.

    Thus, I am all for a transparent ‘what you see is what you get’ review. It may take a bit more effort for reviewers to put in words their criticisms and feelings about a study, but it would make it much easier for the authors to understand the final action of the Editor and to address the central problems of a manuscript for resubmission.

  12. Andreas Meyer-Lindenberg said:

    I have always strongly felt that confidential comments are unhelpful for everyone. As an author, my benefit is greatest when I get maximal feedback from the review process. As a reviewer, my life would perhaps sometimes easier with this option but the extra work required to make the information author-digestible is usually minimal and should be worth it if the information is important for the review process. As an editor, decisionmaking and handling of author inquiries and complaints is easier if the same content is available to all.

  13. Marty Banks said:

    It is inherently unfair to authors when the reviewer sends information that the authors cannot see. If you can’t communicate something to the authors, then you shouldn’t be communicating it to the editor. For this reason, I long ago stop using the confidential comments section. It would be a good move to simply delete it.

  14. Stephen Ikeda said:

    The “confidential comments” section is useful for conveying details of the reviewer’s abilities to the editors. The great breadth of techniques used in many manuscripts means that most of us are knowledgeable about only a portion of the work. Thus our reviews and recommendations are often biased toward the portion of the manuscript for which we feel the most comfortable. The confidential comment sections is useful for conveying this information to the editors so they can make valid judgements about both the review and the manuscript.

  15. Larry Abbott said:

    I think private comments to the editors that the authors cannot see should be avoided. It is extremely difficult for authors to have a paper rejected for reason’s that cannot be understood from the referee reports they receive.

  16. Michael Rogawski said:

    Having served as a member of the PubMed Plus working group that crafted the ground rules for the Neuroscience Peer Review Consortium, I can report that our objective was to avoid any unnecessary incursions on the autonomy, editorial independence or operating procedures of participating journals. Nevertheless, we simply could not get around the fact that a review-sharing scheme would not be fair to authors if they were not made aware of everything that would be shared. I am gratified that there is so much support for the elimination of confidential comments. An unintended consequence of the NPRC experiment may be to hurry along the demise of a peer review custom that seems so problematic for many authors.

    As an executive editor of a consortium journal, I have found that confidential comments—if they are provided—may emphasize a particular area of concern but in substance usually duplicate what is written in the comments-for-authors section of the review, although perhaps less diplomatically. I do not believe that their elimination will substantially impair the editorial process.

  17. Chi-Bin Chien said:

    I confess that I often have put a few sentences in the “comments to the editor” box, but always with care that they match the comments to the author. I have used this mechanism in four ways:

    1) To include a very short bottom-line summary of my review. In cases of very weak papers, this does allow candor to the editor (e.g. “this was terrible”) while presenting the comments to the authors in more diplomatic language.

    I’ve assumed that this might be helpful to the editors, but in the end it’s redundant with the comments to the authors.

    2) To acknowledge limits to my expertise (as Steven Ikeda points out). In general I would be willing to include such acknowledgements in my comments to authors. Things could get sticky if a combative author (and which of us isn’t combative sometimes) starts to argue about the exact competence of an anonymous reviewer.

    3) To comment on other reviewers’ previous comments (in the case of a rereview). When I believe that other reviewers are seriously off-base, it seems useful to be able to say so, and allow the editor to use their judgement. Transparency might be best in this case, but again, could make the editor’s job more difficult.

    4) In cases of potential ethical problems. I think it is also important to mention the issues in the comments to authors, but confidential discussions with the editor seem useful in such a potentially charged situation. As several have pointed out, private emails or phone calls would be another option, but I would favor having a designated box in the web review form in order to preserve an audit trail. I think it’s important to have a clear procedure for such cases, which is clearly set out in the instructions to reviewers.

    Overall, I agree with the overwhelming consensus for eliminating or strictly delimiting the use of confidential comments shown by the comments so far.

  18. Chris Frith said:

    I hardly ever use the box ‘comments for attention of the editor’.

    I would be happy to see this box eliminated.

    Report this comment Cancel report
    Your details

    Please confirm the words below

    In order to reduce spamming, this process ensures you are a real person and not an automated program.

  19. Eric Nestler said:

    I don’t see any real value to the confidential comments section. To be fair, anything the reviewer says to the editors, he/she should also say to the authors.

  20. Rachel Wilson said:

    I don’t think referees should be encouraged to write comments that the authors can’t read. There are sometimes unusual situations where a referee may be justified in wanting to communicate privately with an editor. However, the referee is always free to just e-mail or phone the editor directly. This will always provide a forum for confidential comments, whether we like it or not.

  21. Greg DeAngelis said:

    In the vast majority of cases, I think that confidential comments to the editor cause much more trouble than benefit. It is quite problematic for me, as an Associate Editor at J. Neurophysiology, when a reviewer’s scores and confidential comments do not square with the tone of their comments to the authors. I then have to deal with authors who appeal the decision, and I need to convey some sense of the reviewer’s actual level of enthusiasm without violating the reviewer’s confidence. I think that everyone would be better off if the comments to the authors contained all of the reviewer’s opinions and accurately reflected their level of enthusiasm for publication in the particular journal in question. Of course, reviewers should make these points in a collegial manner, but I think that we will ultimately have better (and much more portable) reviews without confidential comments to the editor. This does represent a departure from the way that many journals have operated in the past, and I think that it will be important to broadcast to potential authors and reviewers that the journals would now like comments to the authors to be complete.

    Clearly, there needs to be some way for a reviewer to communicate to the editor if she/he feels that there may be an ethical issue or some other matter that is particularly sensitive. But there could be a box for this which is clearly labeled such that reviewers will know only to use it under special circumstances. That problem could be easily solved I think.

  22. Zach Mainen said:

    I’m totally in favor of the proposal and forgoing confidential comments. I rarely use them when reviewing and I’ve sometimes been suspicious of them as an author.

    Eliminating them would force reviewers to argue their positions more carefully, worth the sacrifice of making reviewing a bit more complicated.

    Increased transparency along with a streamlined review process—sure sounds like a win-win. I can think of a few more things to fix while we’re at it, but I guess that will have to wait…

  23. Graham Collingridge said:

    As Editor-in-Chief of Neuropharmacology, I immediately welcomed the pilot project initiated by the Society for Neuroscience, for all of the reasons given, and did not hesitate to sign up Neuropharmacology for participation in the pilot scheme.

    As an author there is nothing more frustrating than to have a paper that is considered technically sound and a significant advance (but not quite significant enough in the very subjective eyes of one or two individuals) rejected. When this frustration and waste of time re-packaging the work for another journal is multiplied by the number of neuroscientists worldwide who feel the same (a huge number no doubt) one can see why this proposal is a very good idea indeed. The “excellent but rejected” paper will, more than likely, find an appropriate home more quickly and with much less hassle and frustration.

    Of course the impact factor of the “second choice" journal is likely to be less but impact factor is a divisive influence on the scientific process. What is important is what the scientists think of the actual science, which will be reflected better by the download statistics and, eventually, the citations of the paper concerned (not the average citations of all of the other papers published by that journal over the surrounding 2 year period – note that the influence a given paper has on the impact factor of the journal is, in the vast majority of cases, negligible).

    An additional bonus, of course, is the potentially huge saving in reviewer and, in most cases, editor time. For me its win, win, win (as author, reviewer and editor).

    If this means sacrificing confidential comments then so be it. One can argue the merits and pitfalls of confidential comments until the cows come home but the bottom line is that they will need to go if a journal wishes to be part of the scheme. That has to be the case, because the author must be aware of the information that is being passed from one journal to the next.

    I hope the Nature journals sign up for this scheme for the benefit of the community they serve – the scientists.

  24. Paolo Calabresi said:

    I do not like confidential comments to the editor. I believe that only comments open to the authors allow transparency of the relationship among the editor, the reviewers, and the authors.

  25. Ivana Gadjanski said:

    I think confidential comments are not entirely fair to the authors, since it is a way of giving evaluation and also information on their work, but without letting them know about it. I think it would be better not to have that.

  26. Peter Kind said:

    Just to add to what appears to be a growing consensus – I agree that confidential comments should be removed from the reviewing process.

  27. James E. Goldman said:

    Confidential comments to the editor in most cases reflect the general point of the review and are therefore, for the most part, redundant, and can be omitted. As reviewers, we should be able to communicate our positive and negative views clearly, as well as diplomatically. If we think a paper should be rejected, then we have a responsibility to authors to make our reasons absolutely clear in the review itself. I agree with those responders who feel that there are legitimate reasons for confidential comments – animal issues, human subjects treatment, and scientific misconduct. In those cases, the editor must notify the authors (with a copy to the author’s institution) and should not send reviews to a second journal until the issues are resolved.

  28. John Huguenard said:

    I agree with Christof Koch, that with a bit of editing all relevant comments could and should be included transparently in the comments to the authors.

    Yet there should be a means to communicate to the editors the issues raised by Stephen Ikeda and especially points #2,3, and 4 of Chi-Bin Chien.

    This discussion does raise a related issue, which is that of priority scores and the peer review process. Authors do not typically receive any feedback regarding the “scores” their submission received. While this issue will not likely impact the Peer Review Consortium, it was a source of frustration to me as a reviewing editor for the Journal of Neuroscience. Surprisingly, the comments to the authors were not always consistent with the score given, creating a difficult situation for the editors. Thus while elimination of confidential comments would require reviewers to include all relevant critical comments in the main body of the critique, it will not by itself remove the possibility of a reviewer providing conflicting information, via the priority score, to the editors.

  29. Mike Nitabach said:

    One important issue to be addressed is the use of “scores” that reviewers are asked to give a manuscript in its present state: “highly suitable for publication”, “acceptable for publication”, “more suitable for publication in a more specialized journal”, “not suitable for publication”. This secret scoring system—used by almost every journal I have ever reviewed for—is frequently combined with an exhortation that the comments to the authors not make explicit the reviewer’s ultimate opinion as to suitability of the manuscript for publication. This combination almost always leads me to use the confidential comments to the editor to explain in detail how my “score” would be altered in light of various outcomes of textual revisions and further experiments by the authors.

    After reading the other comments in favor of transparency, it seems to me that there is no reason not to share all this information with the authors. It would, at a minimum, save the time and effort that I know all of us have expended as authors—and some as editors—engaging in e-mail correspondence whose sole purpose is to figure out what the reviewers really think about the suitability of the manuscript, and how much movement is possible given particular textual changes and additional experiments.

    As both a reviewer and author, I see no benefit to keeping these things from authors, and forcing them to engage in mutually coy e-mail exchanges with editors to try to figure them out.

  30. Yi Sun said:

    I think the review process should be as open as possible. I never write secret comments, because I think the authors deserve to be informed with every bits of information that leads to the decision of their manuscript(s).

  31. Margaret M. McCarthy said:

    Everyone keeps talking about transparency and making sure the author has all the information, but almost all of the journals I review for have a series of questions asking about the relative merit of the work using rating scales or canned responses. None of this is ever shared with the author yet plays an improtant role in the decision to accept or reject. How is this different from confidential comments to the editor?

  32. james yue gee said:

    I think that confidential comments are not good for the author.

  33. Dan H said:

    I don’t see the problem with confidential comments. The final accept/reject decision is with the editor. If a confidential comment is inappropriate, then the editor can ignore it. If confidential comments should have been included in the main review, then the editor could theoretically tell the reviewer to adjust or simply make a personalized note of additional concerns.

    I generally use confidential comments as a 2-3 sentence cover letter summarizing my usually much longer review. It contains the same info as the author gets, but might be slightly blunter.

    I’ve also included comments such as “I would reject this manuscript, based on X,Y,&Z as stated in the review, but since the core data and concept is good, if the other reviewers disagree, I’d be glad to re-review” (In that case, the other review was more positive, the editor decided to allow them to make a revision, and I accepted the revision. I don’t think that sentence would be appropriate for the authors, but without it I suspect the editor might have just rejected the manuscript)

    Perhaps I am the exception, but I generally use the confidential section to put things in a more positive light than in the review to the authors.

  34. Karim Nader said:

    Sorry for joining the party a bit late, but I agree with most people here. I try never to use the confidential comments section and don’t see a huge problem removing it. The only time I do is when my review falls between levels of acceptance provided by the website. In that case, I will select the more conservative of the options try to write a note to the editor explaining that the paper was slightly better than that option selected.

  35. Daniel Weinberger said:

    I also believe the whole review process should be open and transparent.

  36. Kim Wallen said:

    I have mixed feelings about confidential comments. I find them useful for letting the editor know my level of enthusiasm about a paper. If I really think that the paper should be published and is really important, that would go in confidential comments, since I really should not communicate that to the author and complicate the editor’s job.

    On the other hand, I have had a paper rejected where it certainly seemed likely that there was something in the confidential comments as the rejection did not fit with the reviews which were very positive with only minor criticisms. I could only think that there were comments directly to the editor about publishing the paper which were not transmitted to me.

    I agree with others, that more transparency would be preferred and we can find other ways to convey level of enthusiasm. If this avoids the misuse of confidential comments then it is a net gain in my opinion.

  37. Dave Moore said:

    I agree with the consensus that confidential comments are generally unnecessary. The one time I do find them useful is when the fixed choice offered to the reviewer about the disposition of the paper (should this be confidential?) does not match an alternate course of action I may wish to suggest.

  38. Angus Nairn said:

    While we seem to be in the minority, I am very happy as a referee and as an Editor to have the availability of confidential comments. The usefulness of these has been nicely summarized by Michael Rogawski and James Yue Gee. Moreover, since as Eric Nestler noted there is still a mechanism (phone call or separate email to the Editor) that can be used for “confidential comments”. So the discussion of this whole issue seems irrelevant.

    Surely it would be better to pass “confidential comments” onto Editors of other journals in the consortium than not to be privy to confidential information passed by phone or email between reviewers and editors.

    Having said this, I have problems with the basic concept of the consortium. As one of a group of senior editors for a journal, we have often built up relationships with our editorial board that allow us to take into account individual levels of expertise. As part of a consortium we would be implicitly asked to accept comments from referees who we may not accept as the “experts” in the field. Moreover, the decision to accept a manuscript is frequently made by Editorial groups who presumably can generate a significant amount of “confidential comments”. Is this material also to be passed along to the next journal in the consortium league.

    One way to avoid the stress on the manuscript reviewing system would be for authors to submit to the most appropriate journal in the first place rather than reaching for the journal with the highest impact factor. Another way would be for reviewers to always write reviews that are complete and give scores/ratings that are consistent with the written comments.

  39. Ted Dawson said:

    I am in favor of full transparency of the review process. As a reviewer I think it is best to let the authors know where you stand on their paper. As an author I would like to know what the reviewers actually think of my paper. Knowing the level of enthusiasm of the referees and their private comments would be very instructive as it would enable you make informed decisions about the suitability of your paper publication for that journal or an equivalent journal.

  40. Action Potential said:

    NN Joins Neuroscience Peer Review Consortium

    When the community is overburdened by peer review, it’s everybody’s problem. As of today, Nature Neuroscience has become part of the solution by joining the Neuroscience Peer Review Consortium, a flexible system that allows voluntary participation by a…