Analysing the purpose of peer review.
Most people accept that peer review is enormously valuable and should be maintained and protected, but few agree on what purpose it serves. Science publishing, even at its simplest, involves complex interactions between researchers (authors), journal editors, reviewers and readers. It is a subtle form of human behaviour that could furnish the raw material for dozens of sociology theses. And when academic endeavour gets mixed up with commercial interests, things get even more complicated.
What does peer review do for authors?
For authors and funding organizations, peer review provides an important veneer of respectability. Findings published in a famous journal take on the aura of the journal’s reputation. This may be well founded — some journals invest a lot of effort in selecting, checking and publishing articles. Yet, even the most-respected journals have been caught out and, despite extensive peer review, have ended up publishing fraudulent or seriously flawed material. If peer review can fail to detect such gross crimes, can we be confident that it picks up other problems? Fiona Godlee and colleagues at the British Medical Journal (BMJ) sent an article containing eight deliberate mistakes in study design, analysis and interpretation to more than 200 of the journal’s regular reviewers, most of whom were aware that they were taking part in an experiment1. The reviewers, on average, reported fewer than two of the errors, which hardly inspires confidence.
How do journals use peer review?
For editors, peer review can help inform the decision-making process. Most editors obtain two or three reviews for each article before deciding whether to publish it. Several studies suggest that reviewer agreement is little better than would occur by chance, and journals would need six reviews for every paper to achieve statistical reliability2. Stephen Lock (a former editor of the BMJ) showed there was little difference between sending papers submitted to a general medical journal out for review and relying on the opinion of a single, experienced editor when it came to judging the papers’ originality, scientific reliability, clinical importance and suitability for the journal3.
Prestigious journals base their reputation on their exclusivity, and take a ‘top-down’ approach, creaming off a tiny proportion of the articles they receive. Rejection rates at The Lancet and the New England Journal of Medicine, for example, are well over 90%. Yet other peer-reviewed journals adopt a different philosophy. Some electronic journals, such as those in the BioMed Central series, operate a ‘bias to publish’, accepting anything that meets minimum scientific and ethical standards (see the article in this web debate by Koonin et al). Between these two extremes are the many speciality journals with rejection rates of around 50%. All use peer review, and may even use the same reviewers, yet the outcomes are clearly different. We therefore cannot regard peer review as a diagnostic test designed to produce a correct answer, as the ‘correctness’ of the outcome depends on journal context. A few, brave journal editors have suggested that peer review should be regarded as a mechanism for deciding where a study gets published rather than whether it gets published3.
What do readers get out of peer review?
For readers with limited time, peer review can act as a filter, reducing the amount they think they ought to read to stay abreast. But readers have different needs. Although some relish the shortest possible articles, those who extract data from publications (for example, for meta-analyses) often bemoan the deficiencies of these publications and the need to contact authors for extra details or clarification. There is some evidence that peer review raises the quality of articles4, but it has not been compared with alternative methods such as electronic templates or more rigorous technical editing with evidence-based checklists.
And, while busy clinicians struggle with information overload, there are increasing calls for major funders, such as drug companies, to publish the findings of all their clinical trials, preferably in a peer-reviewed format. This raises an interesting ethical conundrum: if funders have a responsibility to publish all their findings, but peer review acts as a filter, constantly preventing a certain proportion of articles from being published, is peer review actually distorting the scientific record? If one accepts that there is an ethical responsibility to publish results of all clinical trials and that peer review has a role in this process, that role can only be to improve the reporting of these trials and to determine where they appear, not to determine which will get published. In the past, companies sometimes suppressed unfavourable findings, but this is increasingly viewed as unacceptable, and initiatives such as trial registers are designed to reduce this5. In the past couple of years, some major drug companies have made a public commitment to publish the results of all their trials6. It will be interesting to see whether this leads to a growth in the number or size of journals, or whether companies and their critics will be satisfied with dissemination in non-peer-reviewed forums such as company websites.
Does peer review address the needs of these stakeholders?
With all the different needs and uses of peer review, can we say that it is a decision-making tool of uncertain purpose and/or poor reliability? It acts as a filter designed to remove bad science yet is known to be almost useless, on at least some occasions, at detecting major fraud. Traditional models, where reviewers remain cloaked in anonymity, are also open to abuse, as ideas or data may be stolen from a paper under review, or publication delayed by competitors.
Despite the relatively weak evidence from formal studies, experience tells us that the accuracy, clarity and correctness of scientific papers tend to improve with careful review by colleagues or editors. This is the basis of peer review. As a simple concept it works well, but perhaps we now expect too much of it.
If we want to improve the peer-review process we need to start by agreeing its main purpose. Is it a filter, a distribution system, or a quality-control process? Unless we can answer these questions, attempts at ‘improving’ peer review risk wasting time and effort.
1. Godlee, F., Gale, C. R., Martyn, C. N. J. Am. Med. Assoc. 280, 237-240 (1998).
2. Fletcher, R. H. & Fletcher, S. in Peer Review in Health Sciences 2nd edn (eds Godlee, F. & Jefferson, T.) (BMJ Books, London, 2003).
3. Lock, S. A Difficult Balance. Editorial Peer Review in Medicine (BMJ Publishing, London, 1991).
4. Jefferson, T., Alderson, P., Wager, E. & Davidoff, F. J. Am. Med. Assoc. 287, 2784-2786 (2002).
5. De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C. & Hoey, J. et al. Lancet 364, 911-912 (2004).
6. Dyer O. Br. Med. J. 329, 590 (2004).
Elizabeth Wager is a freelance publications consultant. She is a member of the ethics committees of the BMJ, the World Association of Medical Editors and the Committee On Publication Ethics Council. She has written systematic reviews on peer review and technical editing, and guidelines on good publication practice