Peer-to-Peer

Ethics: Increasing accountability

Kirby Lee and Lisa Bero

What authors, editors and reviewers should do to improve peer review.

Peer review is not currently designed to detect deception, nor does it guarantee the validity of research findings. It should, however, identify flaws in the design, presentation, analysis and interpretation of science and provide prompt, detailed, constructive criticism to improve research.


In order to function well, the journal peer-review system relies on the the integrity and accountability of authors, editors and reviewers. Each may behave unethically in the competitive world of science. Although we do not know how often crimes occur, forms of misconduct could include stealing or suppressing ideas and information, favouritism, and misleading reviews. Financial or personal competing interests can provide the motivation for misconduct. Failure to disclose these has eroded the credibility of scientific research and public trust in the publication process.

We discuss a number of policies and mechanisms that all scientific journals could adopt to improve transparency and promote fair peer review.

Improving transparency

Publish editorial decision-making and peer-review process

Require disclosure of all competing interests

Sanctions for misconduct or failure to disclose competing interests

Open peer review

Peer-review systems and editorial policies differ greatly among scientific journals. Most journals publish detailed instructions for authors submitting manuscripts. But the responsibilities and expectations of authors, editors and reviewers would be clearer if all journals published their policies on editorial decision-making and peer review.

Most medical journals have policies on author competing interests, but less than half have competing-interests policies for editors and reviewers, and only 12% or less publish editor or reviewer disclosures of competing interests (ref. 1). Establishing required, structured reporting for the disclosure of competing interests for all parties, as well as policies for handling conflicts when they arise, would ensure a more equitable and transparent peer-review process.

Disclosure of competing interests leads to enhanced critical appraisal of research (ref. 2), but does not necessarily mean rejection (ref. 3).

We encourage journals to develop tougher sanctions for those who fail to disclose competing interests, like those of Environmental Health Perspectives and the Journal of Thoracic and Cardiovascular Surgery. There is little enforcement of current disclosure policies. Adopting policies that ban authors and their institutions from publishing for some period of time at the journal where the offence occurred would encourage disclosure. Some want to create an international, central blacklist for those who have committed more serious acts of misconduct. But there are legal issues and practical difficulties in investigating allegations of misconduct, in maintaining and enforcing such a list, and in protecting innocent co-authors from punishment.

Open peer review, where the identity of the reviewer is known to the author, allows authors and readers to determine whether the review process has been just. Although the arguments for and against open peer review are discussed by others in this debate, it clearly has a role to play in improving the transparency of peer review.

Promoting fair review

Appeals process

Ombudsman

Ethics review boards

Although many journals have an appeals process for rejected papers, authors may not be aware of such policies. In addition, the process varies widely between journals. Editors and reviewers do make mistakes. We recommend that all journals provide and publish details of a formal appeals process.

A journal could appoint an ombudsman to conduct independent, confidential investigations and give advice. Since 1996, for example, The Lancet has included an ombudsman in their peer-review process4. The presence of an ombudsman could minimize editorial misconduct, provide recourse for authors and reviewers who have been unfairly treated, and help maintain a journal’s integrity and independence.

Some journals have ethics committees (for example, the British Medical Journal) or ethics review boards (for example, the PLoS journals). These clarify and develop existing editorial policies (for instance, on competing interests or confidentiality) and formulate new policies. They also advise editors on ethical questions regarding peer review and authorship, for example, and give guidance on moral responsibilities, such as the handling and reporting of scientific misconduct. The committee usually functions independently, is advisory in nature and publishes its decisions.

In addition, external committees have been formed to offer guidance to editors and improve standards through networking and education. These include the Council of Science Editors, the World Association of Medical Editors and the Committee on Publication Ethics. Although the majority of members and participating journals have a biomedical focus, we encourage editors of all scientific journals to draw on these organizations for resources, advice and support in developing their own policies and procedures.

Facilitating peer review

Trial registration

Data audit

Specific instructions and evaluation tools for reviewers

Rewards for reviewers

Access to data and analyses that are not fully reported in a submitted manuscript may be necessary for thorough peer review. The Council of Science Editors recommends that journals establish data-access policies for editorial evaluation and peer review before and after publication so that the validity of the work can be verified or errors corrected.

Clinical trial registration is one way to promote data access and is now a requirement for publication at many biomedical journals (ref. 5). But, as it currently stands, trial registration does not guarantee access to the raw data and applies only to research involving human subjects. Many practical issues will need to be worked out before we have universal trial registration. Therefore, to apply to all scientific journals, we also recommend the development of policies that allow journals to audit or request any type of raw data for research they believe to be suspect or in need of more thorough peer review.

Although reviewing raw data can be difficult, time-consuming and expensive, having such a policy would hold authors more accountable for the accuracy of their data and potentially reduce scientific fraud or misconduct.

To get the most out of reviewers, journals should provide specific instructions as well as evaluation tools or checklists for assessing certain types of manuscripts. In addition, rewarding peer reviewers may help to increase and maintain a pool of good reviewers (ref. 6). Rewards could take the form of public acknowledgement (a formal letter, published list of reviewers or credit in the final publication), payment, discounted subscription or a continuing medical education credit.

These recommendations are by no means comprehensive, but they would improve accountability of those involved in the peer review process: authors, editors and reviewers. Such changes should help to ensure the integrity of research and the dissemination of research findings in the scientific literature.

REFERENCES:

1. Cooper, R. J., Gupta, M., Wilkes, M. S. & Hoffman, J. R. Conflict of interest disclosure policies and practices of peer-reviewed biomedical journals Proc. 5th Int. Cong. Peer Rev. Biomed. Pub. (Chicago, 2005).

2. Schroter, S., Morris, J., Chaudhry, S., Smith, R. & Barratt, H. BMJ 328, 742-743 (2004).

3. Laine, C., Schaeffer, M. & Stack, C.. Conflicts of interest disclosed by authors of manuscripts submitted to a general medical journal Proc. 5th Int. Cong. Peer Rev. Biomed. Pub. (Chicago, 2005).

4. Horton R. J. Am. Med. Assoc. 280, 298-299 (1998).

5. DeAngelis, C. D. et al. J. Am. Med. Assoc. 292, 1363-1364 (2004).

6. Schaeffer, M., Laine, C. & Stack, C. Continuing medical education credit as an incentive for participation in peer review Proc. 5th Int. Cong. Peer Rev. Biomed. Pub. (Chicago, 2005).

Kirby Lee is a clinical pharmacist with primary interests in evidence-based healthcare, health policy and therapeutic outcomes of chronic disease. His research evaluates editorial decision-making and peer review, assessing the quality and accuracy of health information, and developing interventions to improve the appropriateness of drug prescribing and drug taking in older adults.

Lisa Bero is a pharmacologist with primary interests in how clinical and basic sciences are translated into clinical practice and health policy. She is an adviser to the World Health Organization Drug Action Programme, a member of the Cochrane Collaboration Steering Group and serves on several national and international committees related to research dissemination and conflicts of interest.

Read more See this article in Nature’s web focus here

Comments

  1. Report this comment

    Brent A. Wacaser said:

    I am a graduate student and therefore have only limited experience with the reviewing process. I have however seen some of the problems discussed in the article Ethics: Increasing accountability by

    Kirby Lee and Lisa Bero. I think that it is a very positive debate that has been started.

    One thing that the article didn’t mention, that I believe is worth mentioning is the time factor. It takes time to write a good review. From my experience in watching others, writing a few reviews myself, and in the reviews I have read, I believe that time is one of the major factors that determines a good review from a poor one. Did the reviewer take the time to delve into the report and explore it? You can tell by reading a review about how much effort went into writing it. As an example, I recently read two reviews to a paper submitted to Nature: Materials. The first reviewer basically said in 3-4 sentences published it; it is good. The second had taken some time to give some good well thought out comments and criticism. The first may be nice for authors who, as stated in Nature’s Guide for Authors, “desire quick publication of their novel ideas and approaches.” On the other hand it does not really fulfill the expectations of the referee “to identify flaws, suggest improvements and assess novelty. If the manuscript is deemed important enough to be published in a high visibility journal, referees ensure that it is internally consistent, thereby ferreting out spurious conclusions or clumsy frauds.”

    I believe that scientists have more and more demands on their time. They cannot do everything. They must prioritize. I am afraid that one problem the review process is facing is that many scientists set it at a low priority. I think intellectually few would admit this, but they technically give it a low priority by not giving it time. I think that rewarding the reviewers as mentioned in the article may be a way to increase the priority level. Another is just to increase the awareness of the importance of reviews as is happening in this web debate.

    Brent A. Wacaser

  2. Report this comment

    Brent A. Wacaser said:

    I submitted to the blog earlier, but in discussions with a colleague had another thought. Her comment was about gender inequalities in the process of grant proposal reviews. She said why not just leave the gender and name off the proposal totally. Let them decide on the scientific content. I thought that this could apply to review of scientific papers as well. The authors’ names and affiliations could be removed from the reviewers copy. There are many discussions about conflicts of interest and how to deal with them. This could be one way to avoid prejudgment, both good and bad, of a paper based on the authors’ names and affiliations alone.

    Brent A. Wacaser

  3. Report this comment

    David N. Laband, Robert D. Tollison, David B. South, and Steven S. Ditchkoff said:

    We contend that the problem of unconscionably slow reviews is caused principally by the absence of competition between scientific journals. Scientific journals routinely predicate their willingness to consider a manuscript for publication on the authors’ promise that the research being reported on is not simultaneously under submission elsewhere. Consequently, the journal to which a manuscript has been submitted is a monopsonist – – no other putative ‘buyers’ (journals) are permitted to negotiate simultaneously with the seller (authors). Since he doesn’t have to worry about another journal editor snapping up the manuscript that just landed on his desk, a journal editor has little-to-no incentive to provide a good ‘price’ to the ‘seller,’ in the form of fast initial review or courteous service.

    Permitting simultaneous submissions would, almost certainly, improve editorial handling of scientific manuscripts, including the timeliness of reviews. Assuming that time is valuable to the author (to establish scientific priority, enhance the likelihood of promotion and/or tenure, etc.), competitive journal editors will ‘bid’ for good papers by offering authors speedy reviews and fast time to publication after acceptance. To deliver on these promises, journal editors will have to pay reviewers to provide valuable service in the form of fast, conscientious reviews. Because most reviewers currently are paid little, if anything, they have little incentive to provide good service. This incentive incompatibility can be remedied by linking reviewer compensation to the timeliness and usefulness of the review, as happens on a limited scale in economics.

  4. Report this comment

    Liz Wager said:

    In response to Brent Wacaser’s suggestion, many biomedical journals do remove authors’ names and affiliations before review and some have even tested the effects of such masked review. However, results have been equivocal, showing that it is often difficult to mask authors’ identity effectively (especially in small fields) or failing to show a clear effect of masking on review quality or recommendations. There is a helpful summary of the evidence in the Fletchers’ chapter in Godlee & Jefferson’s book ‘Peer Review in Health Sciences’ (2e, BMJ Books, London).

  5. Report this comment

    Tom DeCoursey said:

    Response to Brent A. Wacaser. I agree in some ways with the idea of removing authors’ names to avoid biasing the review. However, this would be worse than the present (admittedly flawed) system. Labs that produce consistently sloppy and irreproducible work would get a free pass if their names were removed. It is extremely difficult to identify whether a particular study was done carefully or not, but knowing the reputation of the lab helps. The information can work in either direction. Most scientists work their entire career to build a reputation of high-quality research. This track record should be taken into account when their work is reviewed. Conversely, labs that consistently produce quick-and-dirty irreprducible studies should be penalized by having their names attached to their own work!

  6. Report this comment

    Pedro Cintas said:

    Peer review against misconduct: let’s improve small things first

    The ongoing and hot debate on peer review and the introduction of a series of novel trials to handle recent cases of misconduct and fraud could unfortunately lead to unpredictable scenarios, if not a certain paranoia. Peer review was first instituted as an essential element in the advancement of science by improving the content and quality of papers. It is true that peers are best qualified to detect mistakes in research methods and assess the real advance in knowledge within a particular field; but it is extremely difficult to detect good falsifications. Scientists lured by fame or money will certainly be able to put elegant frauds on leading journals or ill-conceived results in others less exigent.

    Remarkably, what peer review could do well is often overlooked. As both author and referee, I sometimes find peer-reviewed articles with important minuses; for instance, schemes or graphs that do not match the text, citations out of context, references that cannot be located (wrong journals or pages, missing authors, etc.). It is clear that neither the referees nor the editorial team (even worse, the authors) did accurately check their text or references. One of my colleagues reports another usual case. After reviewing a paper (which was finally rejected) for a top journal, he noted some key errors in formulas and equations, which were transmitted to the editor in his report. Surprisingly, the paper appeared later in a different and less exigent journal, yet containing the same errors. Why did the authors ignore the expertise of a colleague well familiarized with the subject?

    Why not improving peer review from scratch? Probably, this policy simply requires a touch of commitment and recognition. Then, and only then deliberate misconduct could be an anecdotic part in the history of science.

    Pedro Cintas

    Departamento de Quimica Organica

    Facultad de Ciencias-UEX

    E-06071 Badajoz, Spain

Comments are closed.