Peer-to-Peer

Peer Review and Scientific Consensus

Dr Robert Higgs of the Independent Institute, writes:

Journalists, politicians and advocacy groups refer to “peer-reviewed research” and “scientific consensus” as the authoritative last words on controversial matters involving the natural sciences, from climate change to stem-cell research and genetically engineered foods. But many people have an unrealistic view of how the scientific community actually works.

The peer-review process is not, contrary to popular belief, a nearly flawless system of Olympian scrutiny. Any editor of a peer-reviewed journal who desires to reject or accept a submission can easily do so by choosing appropriate referees.

Unfortunately, personal vendettas, ideological conflicts, professional jealousies, methodological disagreements, sheer self-promotion and irresponsibility are as much part of the scientific world as any other. Peer review cannot ensure that research is correct in its procedures and conclusions. A part of the work in every discipline – from the physical sciences to economics –consists of correcting previous mistakes.

At any given time, “scientific consensus” may exist about various matters. Over time, however, new interpretations, tests or observations may demolish that consensus. For instance, in the mid-1970s, an apparent scientific consensus existed that our planet was about to enter another Ice Age. Drastic proposals, such as exploding hydrogen bombs over polar icecaps to melt them. and damming the Bering Strait to prevent icy waters from entering the Pacific, were put forth by reputable scientists and seriously considered by the US government.

The truth is that scientific research at the upper echelons occurs within a fairly small world. Leading researchers attend the same conferences, belong to the same societies, review one another’s work for funding organizations, and so forth. If you do not belong to this tight fraternity, it becomes extremely difficult to gain a hearing for your work, to publish in a “top” journal, to acquire a government grant, to receive an invitation to participate in a scientific conference, or even to place your grad students in decent positions.

“Scientific consensus” often emerges because the members of this exclusive club, and those who support them, have too much invested in the reigning ideas to let go. In this context, it behooves bright young scientists not to rock the boat by challenging anything fundamental or dear to the hearts of those who constitute review committees of funders or journals. The terms “peer review” and “scientific consensus” often serve to suggest a process of disinterested neutrality and saintly pursuit of truth. Like every other human endeavour, however, science is conducted by people with the full range of human emotions and motives.

Good rules of thumb for the non-scientist might be the following: government-funded research that is used to justify that government’s policy should be suspect, whether or not it’s peer-reviewed; and the research of scientists who appear at press conferences in the company of politicians or activists whose agendas they are there to support should be suspect, whether or not the work upholds the consensus opinion.

Robert Higgs is Senior Fellow in Political Economy at the Independent Institute, editor of the quarterly journal The Independent Review, and the author of Depression War and Cold War, as well as numerous books and more than 100 articles in scholarly journals.

Comments

  1. Report this comment

    Ian Tresman said:

    Many have noted that peer review is “not perfect”, including Professor Brian Martin (“”http://www.uow.edu.au/arts/sts/bmartin/dissent/documents/ss/" rel="nofollow">Suppression Stories“), Juan Miguel Campanario (”Rejecting Nobel class articles and resisting Nobel class discoveries“), and Sophie Petit-Zeman in the Guardian (”Trial by peers comes up short"). Others have noted that “”http://www.eurekalert.org/pub_releases/2004-08/cu-bse081204.php" rel="nofollow">British scientists exclude ‘maverick’ colleagues"

    What perhaps is more of a surprise is that there is no will to better the system. Perhaps it’s time to remove anonymity of referees, and make their reports publicly available.

    It’s one thing to reject a paper because it is not scientifically rigorous, another because the referee disagrees with the findings.

  2. Report this comment

    Dave Smith said:

    Re: Peer review and scientific consensus – Dr Robert Higgs.

    Wow. My first time looking into an NPG blog, (or an NPG anything… referred by a collegue) and what a breath of fresh air, to read such balanced and open commentary on the politics of peer-review.

    Great to see integrity is still alive and well, though it surely has suffered somewhat within some scientific circles.

    Well done Dr. Higgs, and NPG. I’ll certainly spend a little more time looking around here…

    Cheers, Dave Smith, of Oz.

  3. Report this comment

    Steven Ericsson-Zenith said:

    Much of what Dr. Higgs says is true and also well-known. It is difficult to know what to do, except to hope that we as a species become more capable of self-criticism, which is the core of intellectual integrity. Unfortunately, it seems as though we are less good at it today than we were at the beginning of the last century.

    Lee Smolin’s excellent book “The Trouble with Physics” discusses the issue at length. I found his book consistent with my experience and inspiring. I reject government funding, but it is a struggle and many choose to take the easy route.

  4. Report this comment

    Massimo Sandal said:

    I know this will make me unpopular on a NPG blog 🙂 , but shouldn’t be time to ditch the old publication system altogether, peer review included, and move to an arXiv-like open archive of publications? Quality and impact of publications would be, as they are now, a function of citations and scientific community feedback. A paper could still need to be sent to peers and be published with, say, at least three peers comments (and possibly improved on following them), but its publication should NOT strictly depend on peers feedback: these should just serve the function of being a starting point for discussion of the article, and ensure that each paper can benefit of a discussion within the scientific community. What would be wrong for the scientific community with this?

    Maxine replies: Interesting point!

    What would be the motivation of authors to upload (“publish”) their articles in this open way? At present, they gain impact and status from publishing in a particular journal. In an open system, they also run the real risk of being “scooped” — possibly more of a problem in some fields than in others.

    We’ve also had some debate and comments about the motivation of the peer reviewers in an open-feedback system — what are the rewards to them? — and on the career risks to a scientist in openly adding critical comments to a posted paper as part of an online discussion.

    Nature Publishing Group has recently launched Nature Precedings,which is similar to ArXiv but for biology and chemistry. We are certainly interested in exploring the type of pubishing model you outline.

    But at the end of the day, many authors are choosing to submit to journals like Nature. Alternative systems have to be what scientists want to use.

  5. Report this comment

    Steven Ericsson-Zenith said:

    I think there is real merit to the high bar that journals like Nature set. Many of us want to set ourselves against such a bar as part of the self measuring and validation process. Certainly these journals need to review their policies to ensure the best scientific standards free from bias – but it would be a significant loss to the community to loose them. ArVix also has a place and journals need to find a way to work with such prepublication – so I applaud the Nature Precedings effort.

Comments are closed.