Peer-to-Peer

Perspective: ‘I don’t know what to believe’

Tracey Brown

Understanding peer review is key to developing informed opinions about scientific research.

The general public are presented with ‘scientific findings’ from a wide range of sources, some more credible than others. Educators complain that pupils and students use web research with little regard for the status of what they find. Medical helplines are inundated with calls about risks and cures following media stories. And, much to the frustration of scientists, unwarranted scares, pseudoscience and health fads abound. How can judgements be made and useful questions asked?


The public needs to know about peer review as a quality mechanism, which means that scientists should talk about it. But when asked about peer review, most scientists’ minds spring either to the paper they have in the system, or to the papers sitting threateningly in their inbox awaiting a reviewer’s eye, or both. Rarely do scientists stand back from the peer-review system and think of it as a set of principles and expectations that necessitate voluntary submission to the critical scrutiny of peers, and aspire to define what is in the interests of science as a whole (validity, significance and originality as opposed to an unrefereed mish-mash based on position, favour and influence).

Making sense of it all

Some years ago, Sense About Science, the UK education charity for which I work, began to look at how to equip the public with an understanding of peer review. The practices of scientific publishing seemed strangely secret in comparison with what passes for scrutiny and data quality in other parts of society.

We began, through workshops and discussions, by thinking about real questions people ask about the science they encounter, such as ‘Is it true?’ and ‘Does this person know what they’re talking about?’ This led to publication, in 2005, of a guide called ‘I don’t know what to believe’, which made no assumptions regarding people’s prior knowledge about scientific publishing.

Far-reaching uses

The reaction to ‘I don’t know what to believe’ has been overwhelming. Some indication of its uses (‘with my patients’, ‘for my students’) emerged while drafting it, fixing a print run of 15,000 copies. But within a few months, 38,000 had been requested in Britain and a further 20,000 worldwide. The electronic button linking to the online version was appearing on websites covering everything from information for people with Alzheimer’s disease through study guides, community news and mobile phones.

The requests reveal a broad swathe of society involved in passing on scientific information: health workers, librarians, societies, public-health officials, policy-makers, technology companies, safety bodies, popular writers, educators, parenting groups and local government. Medical-research charities responded enthusiastically, as did patient groups, sending out the guide through newsletters. These are the organizations who, when the newspapers move on from a scare story, are left picking up calls from anxious patients and their relatives. Doctors, midwives, nurses and healthcare visitors have also reported borrowing from the guide when confronted with a wave of concern following a television programme on the purported health effects of power lines, vaccines and other sources of contention.

Science reporters in the national media are arguably familiar with scientific publishing, and some struggle hard against the copyeditor’s scissors to keep in the basic details of where research is published. But there are many other journalists, particularly in the local press, who do not have the expertise to interrogate press releases or give suitable weight to different claims, and copies of the guide have been sent to them. Similarly, it is important that press officers and conference organizers know the potential implications of abbreviating or ‘sexing up’ the research they publicize.

More broadly, there seems to be growing recognition of the need for more comment and clarity on the status (published/unpublished/repeated and so on) of scientific findings. In Britain, for example, the guidelines produced by the government’s chief scientific adviser are being rewritten, and the government’s use of evidence is being investigated. And in Australia, the government’s use of peer review for research selection is under discussion.

A discerning public

Telling the public about peer review does more than provide interesting information about how science works. It enables people to develop a more sceptical approach and to question stories for themselves.

Although scientists are beginning to respond, it is still frustrating that many seem to equate telling the public about peer review with promoting false certainties. People can accept that mistakes, delays and poor motives exist in specific cases, while finding it illuminating and useful to have a general rule of thumb (published means scrutinized; unpublished, ask why). They can understand that one paper is not the end of the story and that conflicting views coexist and compete.

The confusion that often surrounds scientific findings, risks and certainties, couldn’t be a better opportunity to embark on a discussion about how science advances and to give people some independence from the daily flow of pseudoscience.

Tracey Brown has been director of Sense About Science since 2002 and is the author of its working party’s report, Peer Review and the Acceptance of New Scientific Ideas (2004). She writes and speaks regularly on the challenges of peer review and on other debates about science, evidence and the public.

Read more See this article in Nature’s web focus here

Comments

  1. Report this comment

    Malcom McLean said:

    The editor directed my attention here.

    No one else is interested enough to comment, which is bitterly disappointing.

    At Leeds University, the caterers have posted a notice saying that “We endeavour to ensure that our meals are GM free, but cannot guarantee no products contain GM ingredients. We are constantly reviewing the situation and are in dialogue with the relevant authorities”.

    If this sort of notice can go up at a university, what are we to think of the level of scientific literacy of the population outside?

    The booklet is a brave attempt to explain the scientific method, but sadly a symptom of the problem rather than part of the solution. The title “I don’t know what to believe” demands a profound work, but the booklet is not that. The idea that philosophical propositions can be subjected to quality control in the same manner as washing machines is alarming.

    It is early autumn in the garden of science. It is hotter than it has ever been, and the branches are laden with produce. But the Sun, which powered this great harvest, is now past its zenith, and, unbeknown to the casual observer, decay has begun to eat at the heart of the fruits.

    A young person entering onto a career in science must know that s/he is like an Arthurian knight defending reason against the pagans. The empire is doomed when our new recruits don’t even know what a logarithm is.

    No one seems to have any interest in providing better, more appropriate advice. We should worry about the sort of thinking that takes hold when commerical companies take over from governments as patrons of the sciences.

    Why not forget the latest grant proposal for once, and try to answer seriously the question, “How do I know what to believe?”

  2. Report this comment

    Peter Martin said:

    I reviewed Sense About Science’s " Peer review and the acceptance of new scientific ideas" report for Chemistry & Industry magazine earlier this year. I found it to be a commendable explanation of the basic process of peer review and what this means for published scientific work. Their follow on guide, “I don’t know what to believe…”, presents the essential aspects of peer review in a popular and versatile format.

    The general public currently has an unprecedented interest in scientific research. This is particularly true in the area of health, but also in other areas – often associated with ethical controversy and safety. This interest is expressed in the numerous reports of scientific developments in the popular media.

    I believe that it is the responsibility of scientists to try and communicate the basics of the scientific method to the general public. This response by the scientific community is necessary in order to ensure that members of the general public can form relatively informed and balanced opinions of scientific reports they encounter.

    If we can succeed in doing this it will both raise the general level of public debate of scientific issues, and it will also ensure that scientists are able to argue rationally for the funding and moral right to do the research we believe is necessary.

    I found the “I don’t know what to believe..” guide to be a great attempt at starting this challenging task.

  3. Report this comment

    Giorgio Pea said:

    I was scrolling through the whole debate on peer reviewing where I could not help noticing how many comments are about “open” versus “closed” systems.

    Many of these arguments address the question of whether or not the reviewers should be known by the authors. But I would like to ask if reviewing processes would allow better accuracy of judgment if authors’ anonymity to the reviewers (or even to the editor) was guaranteed for the first round of peer reviewing?

    Regarding the “costs” of peer reviewing, I would include the time authors have to waste in formatting their manuscripts to the style of the journal, which is particularly frustrating if the work is rejected, before knowing if their work will be considered for publication. In this respect, I think that it would be easier if unformatted manuscripts were to be provided by authors at first submission. Alternatively, all journals could agree to officially recognize a “first-submission manuscript format” (similar to the system for MIAME compliance of microarray data).

Comments are closed.