Josh's blog a long time ago now on how to write a cover letter made me think it might also be useful to provide some thoughts on how to referee a paper - hopefully we can again get some other editors, as well as you guys as authors and referees, to chime in on what is helpful/successful (or not).
First, to clarify what we do before we send the paper to referees: we evaluate whether the manuscript is within the scope of the journal (that is, assuming the results are true, would our journal be the right place to publish it?), whether it's likely to be of broad interest (does it have larger implications, or is it in a big field that everyone is thinking about?), and whether it's likely to be a significant advance (this one is certainly most difficult for us, as non-experts, to evaluate, but if the vast majority of the ideas have already been published, it's probably not a big advance). It is not our job to evaluate the data, although we do consider whether the right data are present (if the author is trying to make a conclusion about a biological process, for example, there usually needs to be biological data in the paper). With the exception perhaps of the first point (journal scope), we do need feedback from our referees on these same issues, in addition to evaluating the data and determining whether the data support the conclusions (and whether they are sufficient to support the conclusions). There is also more information available here.
In addition to those general ideas, there are a few specific tips I thought I would pass along. I'll go from most to least obvious, if I can figure out which ones are which:
1) If you are asked to referee a paper, and are really too busy to take it on, then decline. You may think that the editors will be annoyed with you if you say no, but we find it much more annoying to have to send 8 emails to someone to get them to provide a report that's already 3 weeks late. Even more annoying is when that 8th email still does not produce any response, and we have to either make a decision on the paper without all the facts we need or find someone else, which then delays the process for at least another 2 weeks. A quick, polite 'no' will let us move on to someone else.
2) Similarly, if you do agree to look at a paper, a 3 sentence report ("This paper seems really good") is usually not helpful. However: if you have truly assessed the paper on all conceptual and technical merits within your expertise, and found NO problems at all, it is fine to say something to that effect without belaboring the point.
2b) We identify referees based on their knowledge of some particular field or technique, but are often pleasantly surprised by their comments on an additional aspect of the work. While this is not necessary or standard, I think it would be super cool if referees would tell us in the confidential (editor-only) comments which methodology/techniques they felt qualified to evaluate. This would especially be helpful in cases like #2 above, where no specific comments are provided.
3) Please do not make personal attacks on the author/coauthors. If you have some reason to dislike them on a personal level at the outset, don't agree to review the paper. If you discover in the process of reviewing the manuscript that they have not included the 12 references that make their work completely uninteresting, please just say something like, 'The authors have unfortunately failed to reference several pertinent papers (refs 1, 2, n as necessary). As a result, the novelty of this particular contribution is not clear.' Or if they have missed several important control experiments, please just point them out without suggesting that the authors are stupid or hiding information, etc.. It is possible they are stupid or hiding something, but it's more likely that the experiment was just overlooked. I guess the summary here is: write a report that you would be happy to receive, even if the overall message is a negative one. Similarly, keep in mind that you serve a valuable role not only in assessing the paper, but improving it, which often makes for the most exciting papers we publish.
4) Please use the 'remarks to the editor' field - that's where you can tell us confidentially that you have a paper in press on the same topic, or that you don't feel qualified to comment on a particular aspect of the work; additionally, we often find that referees make their most candid assessment of the manuscript within the confidential comments. However, please do include some measure of your overall assessment of the paper in the remarks to the authors. Our authors can get quite confused when we reject papers because, from what they can see, there were not any problems ('the work seemed well done', etc.) when in reality the referees told us confidentially that the paper was not an advance, not appropriate for the journal, or similar. If you hate it, but are worried about hurting the author's feelings, you could say something like 'I was not convinced that this paper represented an important advance in this field.' After all, it is/should be the authors job to convince you of just that.
5) If you find that there are technical problems, please suggest specific experiments/controls that the authors could/need to do to conclusively prove whatever it is that they are trying to prove. If there are conceptual/logic problems, please suggest alternate hypotheses that would need to be ruled out, or factors they have not taken into consideration, in addition to just noting that the paper is flawed or not conclusive. This is helpful for you, too: If we don't know exactly what the authors need to do to merit another round with the referees, we will be more likely to send the paper back to you (thus taking up more of your time) than if we (and the authors) have better guidelines about how to revise.
6) Finally, please keep in mind that, if you agree to look at a paper, you have signed on for life. It is extremely disruptive for the process if we have to find new referees at the second or third round, etc., both because we have to actually find someone new and because that new referee has a significant extra burden of trying to figure out what the history of the paper is. As I implied above, we do strive to limit the amount of time that referees need to spend with a particular paper, and we do limit the number of times we will contact you with new manuscripts. So unless you have come down with pneumonia or have moved to the arctic circle where no internet is available, please stay active in the process.
Well, that's what you need to know - at least, as far as I can judge. What have I forgotten?
Catherine (associate editor, Nature Chemical Biology)
