Peer-to-Peer

Perspective: Does peer review mean the same to the public as it does to scientists?

John Moore

Even reviewed literature can be cherry-picked to support any argument.

The research community understands that scientific information that has not been peer reviewed should not be taken seriously. As scientists, we discriminate between what is put out on blogs or in press releases and what is published in the formal scientific literature. We also know the difference between a peer-reviewed primary paper or review, and an unreviewed letter to the editor or opinion piece. In other words, we understand the peer-review system, and use it as a filter to sort the wheat from the chaff.


As other contributors to this forum will no doubt opine, the peer-review system has its flaws. My own view is that it’s the least-bad system that can be devised, and that, although it might need tinkering with, its fundamentals should remain intact. One way to consider its operating principle is: ‘Judgement of the scientists, by the scientists, for the people.’ But do the people understand the limitations of the process? I suspect not. Even science writers and journalists who should act as important links between scientists and the public sometimes seem not to appreciate what peer review means.

It’s been peer reviewed, so it must be right, right? Wrong! Not everything in the peer-reviewed literature is correct. Indeed, some of it is downright bad science. Professional scientists usually know how to rate papers within their own fields of expertise (all too often very narrow ones nowadays). We realize that some journals are more stringent than others in what they will accept, and that peer-review standards can unfortunately be too flexible. A lust for profit has arguably led to the appearance of too many journals, and so it can be all too easy to find somewhere that will publish poor-quality work.

The public doesn’t understand this, how could it? But the term ‘peer review’ is often equated with ‘gold standard’. Hence, the politically motivated, lazy or unscrupulous can use the peer-reviewed literature selectively, to make arguments that are seriously flawed, or even damaging to public policy. Chris Mooney, in The Republican War on Science (Basic Books, 2005), provides several examples of how this operates in the political world.

Professional scientists can see through this tactic. We know that scientific truth evolves on the basis of a mounting consensus, not through an isolated paper that adopts a maverick position, even if it has been ‘peer reviewed’. In contrast, politicians all too often cherry-pick the ‘facts’ they find most convenient to their party’s agenda. And politicians are not alone.

In my own field of AIDS research, a small clique of scientists and scientifically ignorant laymen promotes the bizarre view that HIV does not cause AIDS, or, in a particularly dubious variant of the genre, that HIV does not actually exist. These AIDS denialists are experts at selectively using the peer-reviewed literature to superficially bolster their positions. I think they lack the training – or if trained, the integrity – to appreciate two things that are understood by professional scientists. First, that peer-reviewed literature develops over time, so that what was legitimately uncertain 20 years ago is fully understood today. This means that citing decade-old papers and ignoring more recent ones is an unscrupulous tactic. Second, that ignoring every paper bar the one that most conveniently suits a preconceived position could be considered scientific misconduct.

Similar practices can be found in other science-related areas. For example, advertisements claiming that vitamin pills can cure cancer and infectious diseases selectively cite the peer-reviewed literature.

One problem the public faces when trying to understand science is that the peer-reviewed literature is still not generally accessible, despite the efforts of the open-access movement (see ‘Access to the literature’). I am often asked to email scientific papers to AIDS activists who cannot easily access publication databases. I shudder to think how frustrating it must be for the true layperson entering an area of research for the first time, without the professional connections to acquire information, let alone interpret it. The publishers need to wise up; the public has a right to see the papers its tax dollars pay for. Otherwise, the public may resort to the Internet to inform them – to the blogs and websites that all too often promote strange, pseudoscientific ideas.

And so, despite professional scientists laughing at the notion that HIV does not cause AIDS, some vulnerable, newly infected people, who would like to believe that they have not just contracted a deadly virus, end up surfing the web for answers. Sadly, so do some science writers and their editors (see www.aidstruth.org for recent examples). This kind of fiasco might be avoided if the public had better access to the peer-reviewed literature, and if bona fide scientists were willing to give the public more assistance in interpreting it properly.

John Moore is an AIDS researcher now working at the Weill Medical College of Cornell University in New York.

Read more See this article in Nature’s web focus here

Comments

  1. Report this comment

    Enro, scientifique et citoyen said:

    Le peer review en proc賢 />

    En science toute la litt鲡ture est fiable puisque pass饠au crible du peer review ou jugement par les pairs ! Voilࠣe que l’on entend souvent, qui revient ࠡssimiler peer review et sacralisation. Or ce n’est 鶩demment pas aussi simple, notamment…

  2. Report this comment

    Mat Mercur, B.Kin, M.Sc said:

    I was happy to read that I am not alone in my views on the peer review system. One issue brought up “struck a chord” with my current interest in the accessibility to knowledge. Dr Moore is correct in the view that reporting on much of the good work being done is not available to the population at large. What is worse is that much of what is written is done so in an esoteric manner. I feel that using often unnessecary, complicated language in journal reporting exacerbates the issue of misinterpretation by journalists and activists. Unfortunatly, if one writes in a more simplistic manner, one risks not being taken seriously by the peer reviewers (as Dr. John Ralston Saul states: obscurity suggests complexity which suggests importance). What is one to do in such a predicament? I suppose a scientist could find a different form for dissemination of research (i.e. popular literature), but unfortunately, Universities, the scientific community, and granting agencies frown on such activities.

    Mat Mercuri B.Kin, M.Sc, McMaster University, Hamilton, Ontario, Canada

    p.s. In regards to the alluded to Harpers article about AIDS research; I believe that article was about the need for open discourse throughout the scientific community and how activists can hijack that discourse. I find it ironic that many took the specific case scenarios as truth becuase it was published in a reputable magazine.

  3. Report this comment

    Malcolm McLean said:

    If you discuss with members of the public on Usenet you will see that the phrase “peer review” is uttered like a mantra.

    John Moore writes

    “The research community understands that scientific information that has not been peer reviewed should not be taken seriously.”

    What we are seeing is a return to the idea of arguing from authority. As a heuristic it is very good. If the matter is of little real scientific interest of importance then it is the best way, to simply rely on the superior wisdom of others.

    Unfortunately the public believes that significant questions, such as the existence of genetic differences between members of different races, or the cause of homosexual behaviour, can be resolved by recourse to the peer reviewed literarature. The peer reviewers can do no such thing, particularly when the discipline, like psychology, is poorly understood.

    Argument from authority and natural philosophy are not compatible. Despite a massive expansion of the universities, we are now close to the end of the era of rapid scientific progress. With the solitary excpetion of the discovery of the structure of DNA, our world view remains what it was in the first half of the twentieth century. This is not because no significant questions remain – the nature of consciousness, of language, of free will all remain mysterious. Few are working on these questions. The paradigm of twenty-first century science is structural genomics – mass production of snippets of data, or science by robots.

    The British government has recently determined that “a glass of fruit juice counts as one of your five [portion of vegetables you must eat] but further glasses are of no nutritional value and do not count towards it” [quoted in the Times] What is Joe Public to make of this bizarre pronouncement? How is he to understand the limitations of nutritional studies on human populations. We are very close to the end of the age of science.

    The way to drive out bad philosophy, as Nietzsche pointed out, is simply to cease to reward the philsophers for doing it. As soon as publication in peer-reviewed journals ceases to be a passport to comfortable middle-class careers, bad science will cease.

  4. Report this comment

    Maxine said:

    A note from the editor on Dr McLean’s interesting comment: a contribution to this forum to be published later this week on Wednesday 21 June raises some of the issues about public understanding mentioned in this comment, and discusses one approach that attempts to address this (undoubtedly multi-faceted) question.

    Maxine Clarke.

Comments are closed.