Peer-to-Peer

Perspective: Peer review of interdisciplinary scientific papers

Christopher Lee

Boundary-crossing research meets border patrol

Both universities and funding agencies have proclaimed the need for more ‘interdisciplinary’ research and more ‘interdisciplinary’ teams, yet publication of such work raises a unique set of challenges for peer review for which many traditional single-discipline journals are not fully prepared. Many of the most exciting research fields today are themselves hybrids of multiple disciplines, yet the peer review system gives rise to problems in assessing interdisciplinary research. The system can be fixed, given the tremendous opportunities today for new approaches


In this article, I use the term ‘interdisciplinary peer review’ to denote the specific case where a reviewer is not expert in all areas of the submitted paper’s research questions, data or methods. Does this ever happen? Yes, increasingly often. This can occur both for a hybrid study that mixes questions and methods from two previously separate fields, or for a paper that discovers new results in one field using newly developed methods in another (even if some researchers know aspects of both fields, this combination of novel discoveries in one field and novel methods in another can push it over the line, into the problem I’ve defined). Unfortunately, many assumptions of standard peer review break down in these cases.

What are these assumptions? First, reviewers need to have comprehensive authority, to evaluate both impact (results) and validity (methodology) to return a decision. Second, reviewers need to be chosen from the right field for the paper, so that their prosecutorial role (criticizing possible flaws) is naturally balanced by a genuine interest in welcoming innovations within the specialist area.

Assuming this balance, standard peer review needs no checks for the possibility of errors in the claims made by the reviewers’ reports, and controls only for individual variation (by comparing two or more reports) rather than for possible field-bias (in which most reviewers from a given field may be less amenable to evidence from other fields outside their expertise).

Interdisciplinary challenge

In interdisciplinary peer review, as I have narrowly defined it for this article, the reviewer is not knowledgeable about some areas of the paper’s research questions, data or methodology. This can have several effects. First, positive consideration of impact diminishes, while negative consideration of validity can rise to become dominant. Potential reviewers who understand the paper’s data but not the details of its methodology often will decline to review it. Even if they do, their report will often be given less weight by the editor than assessments by the methodology experts. Reviewers who work primarily on methodology but are not knowledgeable about the areas of the paper’s data and results will often criticize it as lacking significance (because these data and results are not of interest to them) or novelty (if the methodology by itself is not very novel) or as invalid (if the actual data in this field require different assumptions than they are used to). It is notable that the most valuable category of interdisciplinary paper for journals and readers (exciting, novel results made possible by novel methodology) may be the most likely to get blocked by this catch-22.

Second, and more important, when reviewers lack thorough knowledge of the detailed literature relevant to these specific data, they can no longer practice the Golden Rule of peer review—“is this work a significant advance over the previous literature?”—and commonly revert to a more conservative question: “is there anything in the paper that seems unsure to me?” While this question might be reasonable if the referee is expert in all aspects of this work, it is often fatal for interdisciplinary peer review. Given that the reviewers are unfamiliar with some areas of the paper, it is likely that they will feel uncertainty or discomfort about some aspects. If reviewers in a given field are more likely to make erroneous claims (statements contradicted by the extant literature) about data that are outside their field of expertise, than about data within their field, this constitutes field-bias. In interdisciplinary peer review, the correct policy is simply to say “I don’t know” in such cases, but under the assumptions of standard peer review, such a statement of ignorance might be perceived as a confession of incompetence as an ‘authority’.

When specialists are asked to review material that goes outside their areas of expertise, standard peer review changes from being a measure of scientific quality to being a measure of reviewer comfort. Journals should consider several obvious fixes:

First assess impact, and afterwards validity. These two phases need different reviewers, from different fields. The impact phase should be fast — a quick search for an audience. If there’s no audience, there’s no impact. Only high-impact papers proceed to validation.

Interdisciplinary peer review requires discussion before decision. Standard peer review requires reviewers to return a judgment as the first step—“shoot first and ask questions later”. But for interdisciplinary peer review, many questions need to be asked and resolved, integrating the expertises of all the reviewers and the authors –before the reviewer can judge the manuscript. This requirement for synthesis during the review process would have been impractical before the Internet, but now is straightforward.

Supported evaluation. Because of the possibility of field bias, interdisciplinary peer review cannot be adjudicated by expressions of opinion, but instead by demonstrated facts from the data, or from the published literature directly relevant to the paper’s data. Because synthesis is required, the other reviewers and the authors need to see specific citations of evidence for a reviewer’s claims, and respond in kind. Above all, reviewers need to state explicitly whether there is an advance on the current literature. Reviewers must distinguish minor false-negative criticism (the results may be incomplete) from major false-positive criticism (the results are wrong). Both authors and other reviewers of a paper can respond to these claims to create a synthesis of the total knowledge of all parties, in the form of data and citations. Both the authors’ and reviewers’ opinions may be changed by this information. Only on this basis can referees make a final validity decision. These criteria are far more important for interdisciplinary than for standard peer review because it is vital to control for the substantial probability of field-bias; and to enable genuine synthesis.

Journals should state a policy on interdisciplinary peer review. Journals that only publish papers within specific, well-populated fields should state clearly that they do not offer support for interdisciplinary research. Journals that do publish boundary-crossing work should publish a detailed policy describing the process, including how they handle each of the issues above. Journals can use interdisciplinary peer review as a separate process applied only to certain papers.

Editors and reviewers should consider field-bias. “I don’t know” should be restored to full honour as a sign of scientific open-mindedness and objectivity. Reviewers should be instructed to confine their comments to their area of expertise, to raise questions before making a judgment, and should be asked explicitly whether they have performed the specific type of analysis used in the manuscript, on the specific type of data it presents. Editors should pay close attention to classic signs of field bias, such as reviews that avoid discussing the paper’s actual data; largely speculative criticisms not supported by specific data or literature; or lack of evidence that the criticisms are actually relevant to the specific data in the paper.

Christopher Lee is in the Department of Chemistry and Biochemistry, University of California at Los Angeles, California 90095-1570, USA. His laboratory is in the Center for Computational Biology focuses on bioinformatics, alternative splicing and genome evolution.

See this article in Nature‘s web focus here

Comments

  1. Victor Pinks II said:

    Christopher Lee lays out the problem well, offering some very insightful solutions. I would like to expand this direction for a moment with more supportive information and new paradigms using the field of computational science for example.

    In the following paragraphs I contend that the current peer review system is not dimensionally capable of addressing inter-disciplinary research. That is, the current peer review system is traditionally vertical along the line of the expert. Using the analogy of a tapestry, vertical research and the current peer review system are single vertical fibers incapable of seeing their place in inter-disciplinary pattern of the whole rug. How does one achieve a clear perspective from that new dimension? Christopher Lee suggests that “I don’t know” be restored to full honor. For inter-disciplinary research, “I don’t know” is scientifically truthful.

    In 1995, The National Academies published a report titled: MATHEMATICAL CHALLENGES FROM THEORETICAL/COMPUTATIONAL CHEMISTRY ( http://www.nap.edu/readingroom/books/mctcc/ ).

    In this study a primary theme emerges – new breakthroughs will come from low risk cross-disciplinary research. The implication being that vertical fields are becoming financially high risk and low impact.

    A few conclusions were: “Many opportunities appear to exist for further collaborations between the mathematical and chemical sciences that could result in high-quality scholarship and research progress that would advance national interests. The productivity of applied computational chemistry would likely be enhanced as a result, which could be potentially significant for industry.”

    “To a large extent, both mathematical scientists and theoretical/computational chemists are relatively unaware of the most exciting recent advances in each others’ fields. Consequently both groups tend to be insensitive to the opportunities for interdisciplinary cross-fertilization that could produce intellectual novelty and productivity enhancements on both sides.”

    Dr. Frank H. Stillinger chaired the committee that wrote that report. He is also one of the founding fathers of the popular molecular dynamics method in computational science. Nearly ten years after that report I asked him if any follow up studies had been made. His reply follows:

    ——————————————————— e-mail start

    —–Original Message—–

    From: Frank H. Stillinger [mailto:fhs@princeton.edu]

    Sent: Friday, October 01, 2004 9:33 PM

    To: Dr. Victor Pinks II

    Subject: Re: NRC Report on Mathematical Challenges from Theoretical/Computational Chemistry

    Dear Dr. Pinks,

    I am not aware of any follow-up report to the 1995 NRC study that I chaired. Although the “Afterword” expressed the hope that the subject would be revisited after passage of a few years, apparently no organization has stepped forward to provide the necessary funding.

    You might be interested to know that the Proceedings of the National Academy of Sciences (PNAS) intends to publish a special issue in the Spring of 2005, comprising contributed papers on leading-edge research in theoretical and computational chemistry. I have suggested to the editors of this issue that the occasion would be appropriate to assess the accuracy and/or impact of the ideas expressed in that 1995 report. I don’t know if this will come to pass.

    Sincerely,

    Frank H. Stillinger

    ——————————————————— e-mail end

    The point here is that the benefits of inter-disciplinary research in computational science have been identified for quite sometime. I expect it is now obvious for most fields of study. It is now such a problem that the very limitations of computational science are constraining scientific discovery (PITAC report – June 2005 https://www.nitrd.gov/pitac/reports/20050609_computational/computational.pdf ).

    There is low risk (known) vertical science and high risk (unknown) vertical science. It looks like we really need to change our risk assessment paradigms along with our inter-disciplinary peer review system: Vertical research is now high risk. Inter-disciplinary research developed from low risk (known) science is now high impact and pioneering. Christopher Lee warns us of the bias that is killing inter-disciplinary research and innovation.

    I would like to offer an additional suggestion to those of Christopher Lee that the principle investigators in any inter-disciplinary research be required to list the fields of expertise needed by the peer review team in order for all vertical elements to be represented. The inter-disciplinary research team must be allowed to request that the peer review committee have certain fields of study represented. Who else would know better?

    Thank you,

    Dr. Victor Pinks II

    Edgebrook Research, LLC at Marmion Academy

    1000 Butterfield Road

    Aurora, Illinois 60178

    Ph (815)739-6785

    vpinks@edgebrookresearch.com

    http://www.edgebrookresearch.com

    Report this comment Cancel report
    Your details

    Please confirm the words below

    In order to reduce spamming, this process ensures you are a real person and not an automated program.

  2. Ian Miller said:

    I believe Lee has described well approaches to address most issues in what Kuhn describes as “normal science”, but it should go further.

    Being able to think outside one’s own discipline is itself a speciality, part of which involves accepting that scientists in different disciplines use different approaches. Not only does a reviewer have to consider material that involves conceptual approaches with which he or she is unfamiliar, but also, material relevant to his discipline will be presented that employs an approach that he/she would not use. If the default position from all specialists is, “I don’t know”, what happens next?

    What to do? Can we find a journal, or an electronic journal, to adopt the position: if the reviewers cannot find anything wrong with it, then it should be available for public discussion, therefore publish?

Comments are closed.