Understanding peer review is key to developing informed opinions about scientific research.
The general public are presented with ‘scientific findings’ from a wide range of sources, some more credible than others. Educators complain that pupils and students use web research with little regard for the status of what they find. Medical helplines are inundated with calls about risks and cures following media stories. And, much to the frustration of scientists, unwarranted scares, pseudoscience and health fads abound. How can judgements be made and useful questions asked?
The public needs to know about peer review as a quality mechanism, which means that scientists should talk about it. But when asked about peer review, most scientists’ minds spring either to the paper they have in the system, or to the papers sitting threateningly in their inbox awaiting a reviewer’s eye, or both. Rarely do scientists stand back from the peer-review system and think of it as a set of principles and expectations that necessitate voluntary submission to the critical scrutiny of peers, and aspire to define what is in the interests of science as a whole (validity, significance and originality as opposed to an unrefereed mish-mash based on position, favour and influence).
Making sense of it all
Some years ago, Sense About Science, the UK education charity for which I work, began to look at how to equip the public with an understanding of peer review. The practices of scientific publishing seemed strangely secret in comparison with what passes for scrutiny and data quality in other parts of society.
We began, through workshops and discussions, by thinking about real questions people ask about the science they encounter, such as ‘Is it true?’ and ‘Does this person know what they’re talking about?’ This led to publication, in 2005, of a guide called ‘I don’t know what to believe’, which made no assumptions regarding people’s prior knowledge about scientific publishing.
The reaction to ‘I don’t know what to believe’ has been overwhelming. Some indication of its uses (‘with my patients’, ‘for my students’) emerged while drafting it, fixing a print run of 15,000 copies. But within a few months, 38,000 had been requested in Britain and a further 20,000 worldwide. The electronic button linking to the online version was appearing on websites covering everything from information for people with Alzheimer’s disease through study guides, community news and mobile phones.
The requests reveal a broad swathe of society involved in passing on scientific information: health workers, librarians, societies, public-health officials, policy-makers, technology companies, safety bodies, popular writers, educators, parenting groups and local government. Medical-research charities responded enthusiastically, as did patient groups, sending out the guide through newsletters. These are the organizations who, when the newspapers move on from a scare story, are left picking up calls from anxious patients and their relatives. Doctors, midwives, nurses and healthcare visitors have also reported borrowing from the guide when confronted with a wave of concern following a television programme on the purported health effects of power lines, vaccines and other sources of contention.
Science reporters in the national media are arguably familiar with scientific publishing, and some struggle hard against the copyeditor’s scissors to keep in the basic details of where research is published. But there are many other journalists, particularly in the local press, who do not have the expertise to interrogate press releases or give suitable weight to different claims, and copies of the guide have been sent to them. Similarly, it is important that press officers and conference organizers know the potential implications of abbreviating or ‘sexing up’ the research they publicize.
More broadly, there seems to be growing recognition of the need for more comment and clarity on the status (published/unpublished/repeated and so on) of scientific findings. In Britain, for example, the guidelines produced by the government’s chief scientific adviser are being rewritten, and the government’s use of evidence is being investigated. And in Australia, the government’s use of peer review for research selection is under discussion.
A discerning public
Telling the public about peer review does more than provide interesting information about how science works. It enables people to develop a more sceptical approach and to question stories for themselves.
Although scientists are beginning to respond, it is still frustrating that many seem to equate telling the public about peer review with promoting false certainties. People can accept that mistakes, delays and poor motives exist in specific cases, while finding it illuminating and useful to have a general rule of thumb (published means scrutinized; unpublished, ask why). They can understand that one paper is not the end of the story and that conflicting views coexist and compete.
The confusion that often surrounds scientific findings, risks and certainties, couldn’t be a better opportunity to embark on a discussion about how science advances and to give people some independence from the daily flow of pseudoscience.
Tracey Brown has been director of Sense About Science since 2002 and is the author of its working party’s report, Peer Review and the Acceptance of New Scientific Ideas (2004). She writes and speaks regularly on the challenges of peer review and on other debates about science, evidence and the public.