Boundary-crossing research meets border patrol
Both universities and funding agencies have proclaimed the need for more ‘interdisciplinary’ research and more ‘interdisciplinary’ teams, yet publication of such work raises a unique set of challenges for peer review for which many traditional single-discipline journals are not fully prepared. Many of the most exciting research fields today are themselves hybrids of multiple disciplines, yet the peer review system gives rise to problems in assessing interdisciplinary research. The system can be fixed, given the tremendous opportunities today for new approaches
In this article, I use the term ‘interdisciplinary peer review’ to denote the specific case where a reviewer is not expert in all areas of the submitted paper’s research questions, data or methods. Does this ever happen? Yes, increasingly often. This can occur both for a hybrid study that mixes questions and methods from two previously separate fields, or for a paper that discovers new results in one field using newly developed methods in another (even if some researchers know aspects of both fields, this combination of novel discoveries in one field and novel methods in another can push it over the line, into the problem I’ve defined). Unfortunately, many assumptions of standard peer review break down in these cases.
What are these assumptions? First, reviewers need to have comprehensive authority, to evaluate both impact (results) and validity (methodology) to return a decision. Second, reviewers need to be chosen from the right field for the paper, so that their prosecutorial role (criticizing possible flaws) is naturally balanced by a genuine interest in welcoming innovations within the specialist area.
Assuming this balance, standard peer review needs no checks for the possibility of errors in the claims made by the reviewers’ reports, and controls only for individual variation (by comparing two or more reports) rather than for possible field-bias (in which most reviewers from a given field may be less amenable to evidence from other fields outside their expertise).
In interdisciplinary peer review, as I have narrowly defined it for this article, the reviewer is not knowledgeable about some areas of the paper’s research questions, data or methodology. This can have several effects. First, positive consideration of impact diminishes, while negative consideration of validity can rise to become dominant. Potential reviewers who understand the paper’s data but not the details of its methodology often will decline to review it. Even if they do, their report will often be given less weight by the editor than assessments by the methodology experts. Reviewers who work primarily on methodology but are not knowledgeable about the areas of the paper’s data and results will often criticize it as lacking significance (because these data and results are not of interest to them) or novelty (if the methodology by itself is not very novel) or as invalid (if the actual data in this field require different assumptions than they are used to). It is notable that the most valuable category of interdisciplinary paper for journals and readers (exciting, novel results made possible by novel methodology) may be the most likely to get blocked by this catch-22.
Second, and more important, when reviewers lack thorough knowledge of the detailed literature relevant to these specific data, they can no longer practice the Golden Rule of peer review—“is this work a significant advance over the previous literature?”—and commonly revert to a more conservative question: “is there anything in the paper that seems unsure to me?” While this question might be reasonable if the referee is expert in all aspects of this work, it is often fatal for interdisciplinary peer review. Given that the reviewers are unfamiliar with some areas of the paper, it is likely that they will feel uncertainty or discomfort about some aspects. If reviewers in a given field are more likely to make erroneous claims (statements contradicted by the extant literature) about data that are outside their field of expertise, than about data within their field, this constitutes field-bias. In interdisciplinary peer review, the correct policy is simply to say “I don’t know” in such cases, but under the assumptions of standard peer review, such a statement of ignorance might be perceived as a confession of incompetence as an ‘authority’.
When specialists are asked to review material that goes outside their areas of expertise, standard peer review changes from being a measure of scientific quality to being a measure of reviewer comfort. Journals should consider several obvious fixes:
First assess impact, and afterwards validity. These two phases need different reviewers, from different fields. The impact phase should be fast — a quick search for an audience. If there’s no audience, there’s no impact. Only high-impact papers proceed to validation.
Interdisciplinary peer review requires discussion before decision. Standard peer review requires reviewers to return a judgment as the first step—“shoot first and ask questions later”. But for interdisciplinary peer review, many questions need to be asked and resolved, integrating the expertises of all the reviewers and the authors –before the reviewer can judge the manuscript. This requirement for synthesis during the review process would have been impractical before the Internet, but now is straightforward.
Supported evaluation. Because of the possibility of field bias, interdisciplinary peer review cannot be adjudicated by expressions of opinion, but instead by demonstrated facts from the data, or from the published literature directly relevant to the paper’s data. Because synthesis is required, the other reviewers and the authors need to see specific citations of evidence for a reviewer’s claims, and respond in kind. Above all, reviewers need to state explicitly whether there is an advance on the current literature. Reviewers must distinguish minor false-negative criticism (the results may be incomplete) from major false-positive criticism (the results are wrong). Both authors and other reviewers of a paper can respond to these claims to create a synthesis of the total knowledge of all parties, in the form of data and citations. Both the authors’ and reviewers’ opinions may be changed by this information. Only on this basis can referees make a final validity decision. These criteria are far more important for interdisciplinary than for standard peer review because it is vital to control for the substantial probability of field-bias; and to enable genuine synthesis.
Journals should state a policy on interdisciplinary peer review. Journals that only publish papers within specific, well-populated fields should state clearly that they do not offer support for interdisciplinary research. Journals that do publish boundary-crossing work should publish a detailed policy describing the process, including how they handle each of the issues above. Journals can use interdisciplinary peer review as a separate process applied only to certain papers.
Editors and reviewers should consider field-bias. “I don’t know” should be restored to full honour as a sign of scientific open-mindedness and objectivity. Reviewers should be instructed to confine their comments to their area of expertise, to raise questions before making a judgment, and should be asked explicitly whether they have performed the specific type of analysis used in the manuscript, on the specific type of data it presents. Editors should pay close attention to classic signs of field bias, such as reviews that avoid discussing the paper’s actual data; largely speculative criticisms not supported by specific data or literature; or lack of evidence that the criticisms are actually relevant to the specific data in the paper.
Christopher Lee is in the Department of Chemistry and Biochemistry, University of California at Los Angeles, California 90095-1570, USA. His laboratory is in the Center for Computational Biology focuses on bioinformatics, alternative splicing and genome evolution.