Bring on the box plots

Box plots are excellent for visualizing important core statistics of sample data. We hope that a new online plotting tool BoxPlotR will help encourage their wider use in basic biological research.

The same three samples plotted by bar chart (left) and box plot (right).

The same three samples plotted by bar chart with s.e.m. error bars (left) and Tukey-style box plot (right). The box plot more clearly represents the underlying data.

A bar chart is often a person’s first choice of plot type when they want to compare values. This is appropriate when the values arise from counting. But when the value is a mean or median of data points taken from a sample, a bar chart is usually inappropriate. As discussed in our March Editorial and the accompanying Points of View and Points of Significance columns, a “mean-and-error” scatter-type plot or a box plot are more appropriate for sampled data. In summary, we strongly recommend that box plots be used when you have at least five data points, but for samples with 3-5 data points mean-and-error plots are more appropriate.

Box plots are heavily used in biomedical research in which statisticians have historically had considerable input into study design and analysis. But although similar types and quantities of sample data also appear in basic research (such as that published in Nature journals) box plots are much less common than bar charts in these manuscripts. Last year in Nature Methods for example, ~80% of sampled data was plotted using bar charts.

Discussions we had with the community suggested that an impediment to using box plots instead of bar charts to graph sample data was due to limited support for box plots in plotting programs commonly used by researchers. It also became apparent that some software that did support the box plot was deficient in communicating to users what the different elements of the plot represented. As a result, strangely labeled box plots were showing up in published papers. At NPG we thought it would be useful to provide authors with a simple online tool they could use to generate basic box plots of their data for publication.

The origin of BoxPlotR
At the VizBi 2013 conference in Cambridge Massachusetts I mentioned NPG’s desire for such a tool at a breakout session chaired by Martin Krzywinski in which the participants, including a young researcher named Jan Wildenhain, discussed what the community needed to create better figures. I also happened to mention our interest in this to Michaela Spitzer while visiting her poster from the Juri Rappsilber and Mike Tyers labs showing how the R-package ‘shiny’ by RStudio can be used to easily convert R code (a popular scripting language for statistics) into a visual application for exploring data.

Later at the conference Jan approached me and said he was intrigued by our desire for someone to design a webtool to create box plots and that he was interested in working on such a project. I happily told him to get in touch with me after the conference so we could discuss it further.

Three weeks after the conference concluded I still hadn’t heard from Jan and was beginning to worry that he had decided not to pursue this. Then… a few days later, I received an email from Jan. Much to my surprise he provided a link to a highly functional tool that he and Michaela, through their own initiative, had gone ahead and created using shiny and R. What followed was a productive and rewarding period of discussion and development during which time Michaela incorporated additional functionality and made selected design changes. The tool appeared so well designed and functional that I encouraged them to submit it to Nature Methods for publication as a Correspondence. After incorporating additional functionality and changes based on comments brought up during peer review BoxplotR was ready for publication.

Sample BoxPlotR plots

Sample BoxPlotR plots. Top: Simple Tukey-style box plot. Bottom: Tukey-style box plot with notches, means (crosses), 83% confidence intervals (gray bars; representative of p=0.05 significance) and n values.

Launch of BoxPlotR
To accompany the publication and launch of BoxPlotR we thought it would be useful to provide some information and practical advice about box plots to our readers. Nils Gehlenberg, a former author of several Points of View articles with Bang Wong, agreed to resurrect that popular column for our February issue with an article on bar charts and box plots. Similarly, Martin Krzywinski and Naomi Altman agreed to delay our planned Points of Significance article on the two-sample and paired t-test and instead devote an article to box plots.

Seeing how the community responded to our interest in creating an online box plot tool and then working with them on this project has been a great experience. This never would have been possible without the initiative and talent of Jan and Michaela or the support they received from their PIs Mike and Juri. We hope both our authors and others find BoxPlotR useful and we encourage feedback. General comments can be made here on our blog or by emailing the journal. For specific bug reports and feature requests please see the contact information at https://boxplot.tyerslab.com.

The dos and don’ts of communicating with editors and reviewers

Some thoughts and advice from the editors at Nature Methods on communicating with us and our reviewers, particularly on matters of disagreement.

In the over nine years that we at Nature Methods have been interacting with authors and reviewers we have experienced a great variety of communication strategies. Some work well…others don’t. In our October Editorial we discuss how neglecting to word criticism productively can undermine the value of the criticism and short-circuit this critical aspect of scientific discourse.

In the three posts that follow we provide practical advice for communicating with editors and reviewers during three critical steps of the publication process. These are: the cover letter, the rebuttal letter and the appeal letter. We hope you find these guides useful and encourage readers to comment on the points made and suggest dos and don’ts of their own.

How to write a cover letter
How to write a rebuttal letter
How to write an appeal letter

Update: It has been suggested that we write a dos and don’ts for reviewers. We agree this could be just as useful for improving the peer review process, possibly more so, and hope to be able to provide this soon.

How to write a cover letter

Part one of our 3-part series on the dos and don’ts of communicating with editors and reviewers.

A good cover letter is a crucial part of the manuscript submission package to Nature Methods. It is not simply an archaic form of communication that is becoming obsolete in a digital world; rather, it should be viewed as an opportunity to convey many important pieces of information about a paper to the editors.

Manuscripts submitted to Nature Methods must first pass an editorial evaluation stage, but as professional editors, we are not experts in every scientific field that the journal covers. Providing context for the paper in a cover letter not only can help the editors reach a quicker decision but also can sometimes tip the balance in favor of sending a borderline paper out for peer review.

Here are some practical tips for potential authors.

The DOs:

  • Do give a brief, largely non-technical summary of the method. Explain how it will have an impact and why the method and its applications will be interesting to a broad biological audience. This can include more forward-looking information about potential future applications that authors may be reticent to share with reviewers or readers of their manuscript. Such a summary is especially crucial for highly technical papers, where the chance that the advance may not be fully appreciated by the editors is often higher.
  • Do put the work in context. Briefly explain the novelty and the specific advances over previous work but be realistic about what the method can and cannot achieve. Many authors are hesitant to compare their work to previous methods for fear that it will appear to reviewers that they are putting down the contributions of other researchers. But editors may not be aware of the nuances of various approaches to address a methodological problem and are more likely to reject a paper without peer review when the advance over previous work is not clear. Authors should not hesitate to discuss freely in the cover letter why they believe method is an advance (most ideally, backed up with strong performance characteristics in the manuscript!).
  • Do suggest referees. If the editors decide to send the paper for peer review, providing a list of potential referees, their email addresses, and a very short description of their expertise, can help the editor assign referees more rapidly. Of course, whether the editor decides to use any of the suggested referees is up to him or her. This is also the place to list researchers that you believe should be excluded from reviewing the paper. (Please note that the names of excluded reviewers should also be included in the relevant field of the online submission form.) The editors will honor your exclusion list as long as you don’t exclude more than five people; if you exclude everyone relevant in a scientific field such that the review process will not be productive or fair, the editor may ask you to shorten the list.
  • Do tell us about any related work from your group under consideration or in press elsewhere. Explain how it relates, and include copies of the related manuscripts with your submission.
  • Do mention any unusual circumstances. For example, known competition with another group’s paper, co-submission to Nature Methods planned with another group, or co-submission of a related results paper to another NPG journal, etc.
  • Do mention if you have previously discussed the work with an editor. As editors, we meet a lot of researchers at conferences and lab visits and many papers are pitched to us. A brief mention of when and where such a conversation occurred can help jog the memory of why we invited the authors to submit it in the first place.

The DON’Ts:

  • Don’t simply reiterate that you have submitted a paper to us and/or copy and paste the title and abstract of the paper. The cover letter should be viewed as an opportunity to present useful meta-information about the paper, and not tossed off simply as a submission requirement.
  • Don’t go on for pages about what the paper is about and summarize all of your results. The editor will always read the paper itself so long cover letters are usually redundant. A one-page cover letter in almost all cases is sufficient.
  • Don’t use highly technical jargon and acronyms. Explaining the advance in a general manner can go a long way in helping the editors reach a quicker decision; cover letters that are largely unreadable are of no help to the editors.
  • Don’t overhype or over-interpret. While a description of why the method will advance the field is definitely appreciated, obvious overstatements about the impact or reach of the work do not help and can even reflect poorly on the authors’ judgment of the needs of a field.
  • Don’t assume that going on about your scientific reputation or endorsements from others in the field will sway us. This is not pertinent to our editorial decision. Our decisions are based on whether we think the paper will be a good editorial fit for the journal, not on the laurels of the authors or because someone important in the field suggested that they submit the work to Nature Methods

And finally, a minor editorial pet peeve:

  • Don’t address your cover letter to “Dear Sir.” This is antiquated language, not to mention often incorrect, given that two-thirds of Nature Methods’ editors are women. Stick to the gender-neutral “Dear Editor” in cases where you are not addressing a specific editor.

Don’t miss parts 2 and 3 of this series of posts covering rebuttal letters and appeal letters. We encourage questions, comments and feedback below. The editors will do their best to answer any questions you have.

How to write a rebuttal letter

A well written rebuttal letter is critical in any resubmission. 

Once the initial reaction, be that joy, anger or frustration,  to receiving feedback from editors and reviewers about one’s work has subsided, it’s time for our authors to make one of two decisions:  continue to go after a Nature Method paper  or take their work to another journal.

A realistic look at how the reviewers’ requests can be met will go a long way in helping to determine whether a revision is likely successful and to avoid a futile resubmission.

If authors want to resubmit in cases where the editorial decision was negative, and referees were critical and asked for a lot of additional information, the first step, before embarking on any revision, should be an appeal (see the post on “How to write an appeal letter” for more details) and rebuttal letter to the editor to discuss whether a proposed list of additional information is likely to address the referees concerns.

Authors who receive a positive editorial decision and who are confident that they can address the reviewers’ points nevertheless have to submit a rebuttal letter with their revision.

The rebuttal letter is an author’s chance to directly reply to the reviewers, announce plans to improve the work, clear up misunderstandings or defend aspects of the work. How it is written can make a big difference in whether or not an appeal is granted and how the reviewers judge the revision.

The DOs:

  • Do acknowledge that the reviewers spent a substantial amount of time looking over the paper – rebuttal letters that thank the referees for their time and comments set a positive tone and ensure that the exchange takes place on a productive footing.
  • Do acknowledge that a misunderstanding may be due to poor presentation on your part, not lack of expertise on the reviewers’,  and phrase your reply accordingly,  taking the opportunity to clarify.
  • Do copy the full text of each reviewer’s comments in your rebuttal and reply to every concern raised by each reviewer immediately after each point in a concise manner that clearly states how you plan to address it (experimentally or editorially) or point to data that already addresses it which the reviewer appears to have missed.
  • If you cannot address a point at all, explain why not.
  • Do number the comments or at least break them into paragraphs, and use different fonts or text colors to distinguish the reviewer comments and your reply, rather than write a single reply to an entire review in summary form.
  • Do include relevant citations with full references or dois so they can be easily looked up, rather than just cite by First Author et al.
  • Do include pertinent new data as embedded figures, tables, or attachments,   indicate where in the manuscript you added the information; give page numbers, figure panels, Supplementary material etc., so editors and reviewers don’t have to go on a search for the new data. If any of this information will not be included in the revised paper explain why not.
  • Do be succinct and to the point and avoid epic discourses.  In the case where more than one referee has raised the same concern, it’s best to cite “see response to point 2 from Reviewer #1”, for example.
  • Do remember that each reviewer sees all comments and your replies so be equally respectful to all.

The DON’Ts:

  • Don’t vent or accuse the reviewers of bias or incompetence. We have read countless times that “ ref 2 is lacking expertise and completely misses the point” etc. and one wonders what the goal of such blanket statements is. They serve no productive purpose and instead potentially bias all referees, even the positive ones, against the work.
  • Don’t plead that for personal or monetary reasons critically important experiments can’t be performed. While we hear the plight of underfunded labs we don’t make exceptions for these reasons.
  • Don’t ignore specific requests by referees without comment and selectively only answer a few queries.
  • Don’t rephrase a referees’ point to give it a slightly different meaning that you can more easily address.

Don’t miss parts 1 and 3 of this series of posts covering cover letters and appeal letters. We encourage questions, comments and feedback below. The editors will do their best to answer any questions you have.

How to write an appeal letter

Although usually unsuccessful, a strong appeal letter can be an important tool for authors.

Rejection is never easy. You’ve put long weeks, months, maybe even years of work into a project that you think is perfect to publish in Nature Methods, so your feelings of disappointment, anger, frustration or self-doubt are completely understandable. Your first instinct might be to hit “reply” and send an angry email to the editor. But your best bet is to take some time to cool off, then move on and submit the paper elsewhere. If you are convinced, however, that a serious error has been made or that you can fully address the specific criticisms raised by the editors or by referees, then you may send a constructive appeal letter to the editor.

Nature Methods has different types of rejections, with or without peer review. There are outright rejections (which represent the vast majority), and then there are those rejections where the editor indicates that a manuscript could be reconsidered if the authors can address specific shortcomings. The editorial decision process is of course a subjective and imperfect one. Appeals, however, are usually unsuccessful. Those that are successful are those where the authors make a strong case for reconsideration, typically by proposing to add new data that will strengthen the application or demonstrate how the work is a strong advance over existing methods.

Be aware that appeals are necessarily given a lower priority than manuscripts still under consideration. Decisions on appeals can therefore take a considerable amount of time and the majority of appeals are turned down. Decisions are usually only reversed if the editors can be convinced that the decision to reject was a serious mistake, if the authors can add a substantial amount of data to address certain shortcomings, or if a negative referee is found to have made serious errors or show specific evidence of bias.

An appeal letter is not the same as a rebuttal letter to referees (see the related post, “How to write a rebuttal letter”). An appeal letter is only read by the editors, so sensitive information not meant to be seen by the referees can be included.

Here are some things that do and don’t work when writing an appeal letter.

The DOs:

  • Do consider whether you have a good case for appealing that is worth investing time in the process. By editorial policy, appeals must take second place to new submissions. This means that it can take as long as several weeks for the editors to discuss an appeal, possibly get input from referees, and reach a decision. Unless your case is very strong, it will save you precious time by accepting the editorial decision and submitting the manuscript elsewhere.
  • Do clearly explain the reasons why you disagree with the decision to reject. In some successful cases, authors provide new information, not apparent from the original submission, explaining how the method will have a strong impact on a broad audience. Ideally such information would have been included in the cover letter with the original submission (see the post, “How to write a cover letter”), which can help avoid the need for a lengthy appeals process for a manuscript that is otherwise a good candidate for peer review.
  • Do explain how you plan to rectify any major shortcomings pointed out by the editor or by the referees. If you are willing to add data to the paper to address the shortcomings, explain what this data is and what it shows. If you have figures or tables prepared, include them with your appeal letter. (However, don’t yet rewrite your manuscript – since most appeals are turned down, this is usually just a waste of your time.) If you have a valid reason for not including such data, explain why not. 
  • Do include a separate point-by-point rebuttal letter to referees to assist the editors in reaching a decision (see the post on “How to write a rebuttal letter”). If the editors feel that a rebuttal letter is required to help them reach a decision, and one is not included, they will request one of you.
  • Do provide evidence for any accusations of referee bias. Describe in specific terms why you believe a referee is biased or has made technical errors in their review. In our experience, it is extremely rare that ALL of the referees of a paper would be biased or misjudge its impact. Don’t try to guess who the referees were (you will most often be wrong). In cases where one set of negative referee comments is far out of line with others that are generally positive, we often will consult with the positive referees to determine whether the dissenting referees’ concerns are serious and how they should be addressed.

The DON’Ts:

  • Don’t do anything in the heat of the moment but take some time to cool down and consider whether you would be better off resubmitting elsewhere.
  • Don’t simply reaffirm the importance of the work, write “we think you are making a mistake” or urge us to send a manuscript out for peer review without providing any justification. Appeal letters lacking a good justification will not convince us to change our minds.
  • Don’t try to bribe us with promises of high citations. While of course we hope for high citations for each research paper we publish, citation potential is far from being the most important editorial consideration (and it cannot be accurately predicted, anyway). Papers must first meet our standards of methodological novelty and potential community interest and impact.
  • Don’t assume that the paper must be of interest to us because we have previously published a similar paper. Editorial standards are constantly evolving, and the methodological novelty may be compromised by our previous publication. Additionally, we strive to publish a variety of novel methods across fields, so we must consider what is currently in our pipeline.
  • Don’t bash previous work. As editors, we want to know how a new method addresses certain shortcomings or significantly expands the applicability of a previously published method, but this discussion should be fair and balanced. Don’t simply say “the previous method doesn’t work,” explain why, and ideally provide experimental evidence. Providing a detailed comparison to previous methods in your paper in the first place can help avoid the need for a lengthy appeals process for a manuscript that is otherwise a good candidate for peer review at Nature Methods.
  • Don’t expect us to be swayed by your scientific reputation. While it is informative to give some background of your expertise in a field, we make decisions based on the fit of a paper with our journal in terms of scope, novelty and potential impact, not simply because the work comes from a good lab. The fact that you have coauthored papers in high-impact journals will not lead us to reverse our decision.
  •  Don’t rely on “celebrity endorsements”. It is good to hear that a leader in the field has read and likes your paper, or that 50 people came to view your poster at a conference. But if we feel that the paper is editorially not a good fit for Nature Methods, this is unlikely to make us change our minds about rejection.
  •  Don’t insult the intelligence or competence of the editors or referees. We know that rejections are upsetting and can often seem unfair. But personal attacks and bullying could compromise your success in an otherwise promising appeal situation.
  •  Don’t appeal every decision. Remember the old saying, “you’ve got to pick your battles.”

Don’t miss parts 1 and 2 of this series of posts covering cover letters and rebuttal letters. We encourage questions, comments and feedback below. The editors will do their best to answer any questions you have.