Nature Genetics | Free Association

In defense of peer review

Obviously it will take some time for the dust to settle from the stem cell fraud case. It’s certainly disheartening to see that such an apparently promising result was faked, and no doubt this plays into the hands of those who find the entire endeavor disreputable. But the work will go on, and I completely agree with the bloggers at the American Journal of Bioethics (who have been all over this story) that the question of whether the US federal government will fund this work becomes less and less relevant. The states are the big players, and the Hwang fiasco may actually jumpstart some new work now that the key task of generating individualized embryonic stem cell lines has been put back on the table.

That said, I don’t have anything else to add to the avalanche of commentary that is already out there. I would like to say something about peer review, however, which takes a beating whenever a bad apple turns up in the published literature. Everyone likes Winston Churchill’s line about democracy: it’s the worst form of government except all those other forms that have been tried from time to time. The same goes for peer review, they say (see here). This is appealingly modest, and properly recognizes that all human institutions will be imperfect because all human beings are imperfect.

But this strikes me as damning something with faint praise. So let me go out on a limb and say something heretical: peer review really works quite well.

Really.

Phil Campbell, editor of Nature, makes a strong case, but here are some additional thoughts.

From the questions I sometimes get, one might think that the peer review system is riddled by unacknowledged conflicts of interest, and inadequate or incompetent reviewing. Add in a general tendency to give sexy results a free pass or, depending on whom you’re talking to, to disregard truly important findings out of a lack of imagination, and it’s not a pretty picture. Do such things ever happen? Sure (see above re: human imperfection). But I’ve been an editor at Nature Genetics for more than five years now, and if I had to offer one overriding impression of peer review it would be the conscientiousness of the vast majority of reviewers and editors that I’ve worked with (this may seem self-serving, but there you go). Our reviewers are among the busiest people you’ll ever meet, and yet they labor over the details of manuscripts, considering them in the context of the prior literature, and offer thorough advice on how to improve them, even if they don’t think a particular paper is a good fit for the journal to which it’s been submitted. On occasion, they suggest that a result is flat-out incorrect. Their advice is often lengthy and remarkably thoughtful. As a result, the scientific literature as a whole is significantly better than it might otherwise be: important controls are added, stories are made more complete, new and interesting experiments are carried out, discussions are made clearer, methods are fleshed out. I know that at least one of these things is true of most papers that go through the process.

Perhaps you have to wade through hundreds or thousands of reviews to get a sense of this. Planes that take off and land safely don’t make the headlines.

I’m sure there are a lot of opinions about this. The floor is yours.

Comments

  1. Report this comment

    Roland Krause said:

    How do societies react to offenses against them? No one wants to live in a society that would make it impossible to commit any crime. Likewise, a publishing process that would make it impossible to publish fabricated results would not be favorable to scientific progress. I don’t agree that everything worked well. Journals should review their procedures and must learn from the events – but there is no fundamental flaw in the peer review system.

  2. Report this comment

    patrick sulem said:

    I like to see the peer review as an equivalent of the phase III of clinical trials. They do happen in a standardized manner, and concerns a limited experiment. In the same comparaison, publication of an article is similar to a drug licensing. And then the main play begins, phase IV being the follow up of efficacy and safety in population of at least a 1000 times more users ; and feedback on articles being the follow up on a population of up to a 1000 times more readers.

    In both cases of drugs and papers , the last step is key but is much less obvious to standardize. Like report of side effects or inefficacy in unselected population are key to the drug life, “comments” ,“confirmation or not” and “reactions” to published material are key to the life of articles. The case of fraud does not have to be separated from publication of biased or incorrect material when even the writer is sincere , the veracity of the conclusions is not present.

  3. Report this comment

    Chris said:

    We see today [http://www.nature.com/news/2006/060116/full/439248b.html] that “A Norwegian researcher dreamed up the lives and lifestyles of some 900 people – and used them in a study on cancer. Then, last October, Jon Sudbø had his results published in The Lancet.”

    I completely agree with Alan that reviewers are generally the nicest people around, and work very hard at a task which gets them few tangible benefits. I also think we have to conclude that if someone wants to commit fraud, and they really put their mind to figuring out how to do it [i.e. making up 900 patients!], about the only way to catch it is when it fails to replicate later, or someone affiliated with the lab comes forward.