The Niche

George Daley: Although I think it is always healthy to undergo self-critical soul-searching, I do not see any reason to single out cloning papers for extra scrutiny. Cloning papers are likely subjected to extra scrutiny anyway because of the natural tendency we all have to challenge bold claims. In the aftermath of Hwang, cloning papers will certainly have to meet an incredibly high standard in the future.

Nature: Is Nature's current review procedure adequate as it stands for refereeing of cloning papers?

GD: I think the refereeing process is adequate, but that review of raw data and assessment of internal replicability might be expanded.

Nature: If not, what improvements are needed?

GD: See answer to first question.

Nature: Would the peer review process of cloning papers be improved were Nature to establish a checklist of minimal standard criteria for authors and referees to refer to during the peer review process?

GD: I think the quality of the review is directly related to the expertise of the reviewers and the amount of time they have available to render a detailed critique. We often are so busy that we can give only a cursory glance at a paper. When a paper is clearly destined to be high profile, then extra attention should be accorded to finding the right range of reviewers, and they should be given adequate time to do a thorough and painstaking review.

Nature: If yes, what sort of experimental data should be considered and to what extent do the raw data need to be presented to reviewers, and to what extent should it be published?

GD: I think editors could think creatively about expanded access to raw data. This is happening anyway because much more data can be published as supplemental material on line. Given that this means less pressure to distill data to its essence to conserve copy (which subjects data to selection bias) then a broader representation of the supporting background data can be presented to the readers. Perhaps more attention should be paid to the number of replicates per experiment, and the range of experimental results obtained.

Nature: Last year, Nature asked for independent verification of the Hwang paper showing that Snuppy is a cloned dog. Do you feel this kind of independent verification should be requested for all cloning papers? If yes, what should it entail?

GD: I cannot imagine who would pay for and perform such independent replication. A simple genotype is one thing--a complete experimental replication is another. During the review process, this might be encouraged for particularly bold claims--but the paper must not have immediate competition for priority. It would have to be requested only in unusual cases.

There clearly have been some papers in the recent past that might have benefited from extra measures to find corroborative data before publication. As cases in point, I think of the Tilly papers, which is destined to be challenged by a flurry of contradictory papers.

Nature: Aside from the 2004 and 2005 Science papers from Hwang's group, do other flawed cloning papers exist in the literature? If yes, please identify them and briefly highlight problems and how they could be rectified or prevented by the peer review process.

GD: There are lots of papers that are difficult to replicate, but this does nto always mean they are wrong or fabricated. It can sometimes mean that the techniques are challenging and not yet optimized. This of course makes them less valuable to the community, but nonetheless the proof of principle established by the experiments may be highly valuable.

Replication ensures that a scientific result is durable. Replication also means that an experimental process is understood well enough under controlled experimental conditions that it can be replicated and extended by others. When a paper reports a critical technical "first", then the easiest way to ensure that the paper is robust is to probe deeply into raw data to determine the efficiency of replication within the original lab. If that internal replication is not robust, then the result might be correct but the process clearly has not been optimized.

Nature: Is Nature's current review procedure adequate as it stands for refereeing of cloning papers?

Shin-Ichi Nishikawa: Yes, for peer review. But, because of their popularity and status, prestigious journals like Nature may be required to develop
a different review system particularly for avoiding the possibility of fabrication.

Nature: If not, what improvements are needed?

S -I N: One possibility is to ask reviewers to accept the possibility that there may be a situation where the whole review process including the reviewers’ names need to be disclosed. How reviewers found the study during the review process, including the reviewers’ dialogue with authors, would be useful to gain some understanding by the public.

Nature: Would the peer review process of cloning papers be improved were Nature to establish a checklist of minimal standard criteria for
authors and referees to refer to during the peer review process?

S -I N: For animal experiments, I do not think it necessary to have such a check list. In contrast, for human experiments, we need to deal with difficult questions such as the privacy of donors of materials used for experiments. The most important issue for any science papers is reproducibility. This can be attained only by disclosing all materials and methods such that any other researchers can repeat the experiments. However, in the case of human experiments, many samples used are made anonymous. Of course, it is possible to ask researchers to keep samples for future tests, but it may not be easy to assure with 100% certainty that the samples are kept properly. Indeed, I heard that Dr. Hwang is claiming that he is a victim of conspiracy. If the origin of samples is not traceable, we cannot be 100% sure. Thus, a check list may have some psychological effect, but otherwise may not be an absolute solution.

Nature: We have recently asked for independent verification of the Hwang paper showing that Snuppy is a cloned dog. Do you feel this kind of independent verification should be requested for all cloning papers? If yes, what should it entail?

S-I N: No. I do not think it necessary.

Nature: Is Nature's current review procedure adequate as it stands for refereeing of cloning papers? If not, what improvements are needed?

Teruhiko Wakayama: Cloning experiments require a lot of technical skill. Usually, a postdoc or student will do the experiment, whereas the Professor has little hands-on experience. I cannot imagine that such a professor can review and judge such technical papers, especially in the case of Hwang's report, which was only a technical improvement. In such case, I think the most suitable referee is skilled postdoc, not famous professor. I hope journal editors will ask not only top scientists, but also young scientists, to review papers.

Nature: Would the peer review process of cloning papers be improved were Nature to establish a checklist of minimal standard criteria for authors and referees to refer to during the peer review process?

TW: Yes, good idea, perhaps no only for cloning paper, but also any other technical reports.

Nature: If yes, what sort of experimental data should be considered and to what extent do the raw data need to be presented to reviewers, and to what extent should it be published?

TW: The editors should ask to see the author's CV, to see how many related paper were published, to help them assess whether they have good technical expertise or not. The authors should show ask exactly how many oocytes are used (e.g. the number of corrected normal oocytes, number of lost oocytes by technical accidents etc.) Usually these data are omitted in the paper to make clear. However, these data can be easy to judge the authors technical skill. It is not necessary to publish them, only for the referees to judge.

Nature: Last year, Nature asked for independent verification of the Hwang paper showing that Snuppy is a cloned dog. Do you feel this kind of independent verification should be requested for all cloning papers? If yes, what should it entail?

TW: No. it is impossible. For the completely novel technical paper, at the time it is submitted, nobody knows this new technique and therefore others must learn and gain enough skill to repeat it in order to verify the results. It may take a long time. If it is easy to do, then it is likely that someone else already has published a similar paper.
I think, for cloning papers, that the author should submit all data, or as much as possible. Even mistakes that occur during “trial and error” type experiments. “Data no shown" should not be permitted for this type paper. Independent verification would only cause cost and delays.

I understand your concern, but the simple truth is anyone can lie/cheat, and make up data. There is really no way I can see that you can improve on scrutinizing cloning papers or any other papers for that matter.

Every cloning paper needs to have some documentation that the animal is indeed a clone i.e. simple genetic analysis. But again, this could be manufactured so what's the point of overthinking this?

All one can really do is look at the information/data submitted, send it out for peer review, and make a decision on it, as you have done in the past and as other journals do. With the technology that is out there today to manipulate photographs, data, etc, you basically just have to trust that the investigators submitting the manuscripts are being truthful and ethical, as there is really no way you can control this aspect.

I am a pretty firm believer in "what goes around comes around”. In the end, the reproducibility of an experiment or observation will draw attention to the authenticity of published work.

In brief, I think it is admirable that you and the rest of the group at Nature are evaluating your process for reviewing and scrutinizing cloning papers.......but don't waste a bunch of time beating yourself up over this. You can not control ethical behaviour, and if people want to lie, cheat, and steal they are going to do so, and there is not much you can do about it. I can not see this as much different than going to the mall to buy something, having an outlaw knocking you over the head and taking your money......then you waste time thinking?--perhaps I shouldn't have gone to the mall......or if so......I shouldn't have brought my money?

Nature: Is Nature's current review procedure adequate as it stands for refereeing of cloning papers?

RL; Yes, I think your current review procedure is rigorous and fair. However, I think the key is having good editors who have the ability to select fair and unbiased reviewers. As you know, finding objective referees can be difficult - the old boy network is healthy and strong in the field of cloning. I've been at this for many years, and can honestly say I'm not surprised the Hwang papers got through the review process.

Nature: If not, what improvements are needed?

RL: I'd be VERY cautious about implementing too many improvements. I don't think your review process is broke. Hwang was an extremely dishonest person. I'm in favor of independent or additional testing whenever it's possible, especially when it can be done easily and in a timely fashion. However, this may not always be scientifically possible, so you need to have the flexibility to consider other factors. Whenever there may be uncertainty, I think it would be entirely appropriate to request the work and data leading up to the discovery. Breakthroughs rarely happen in a vacuum. Even within my own lab, I routinely ask scientists in other groups to crosscheck other scientist's data, experiments, and results.

Nature: Would the peer review process of cloning papers be improved were Nature to establish a checklist of minimal standard criteria for authors and referees to refer to during the peer review process?

RL: Again, I don't think the problem lies with the review process itself, but rather with procedures that will help prevent another fiasco like Hwang. If you ask me, I think the next disaster could be accidental. For instance, we received some canine ES cells from a lab that seemed to grow too well - we ran some studies and found they had been contaminated with mouse ES cells. All it takes is for one person to grab a wrong dish from the incubator, deliberately or not.

Nature: If yes, what sort of experimental data should be considered and to what extent do the raw data need to be presented to reviewers, and to what extent should it be published?

RL: For breakthroughs involving ES cells, I think DNA fingerprints should be routinely furnished. For instance, it could be quite embarrassing if a human ES cell line derived using some unique procedure - SCNT or otherwise - had the same fingerprint as H1 or H9. In the future, we plan to fingerprint ALL our human ES cells, to rule out errors or accidental contamination at any point.

Nature: Last year, Nature asked for independent verification of the Hwang paper showing that Snuppy is a cloned dog. Do you feel this kind of independent verification should be requested for all cloning papers? If yes, what should it entail?

RL: Yes, as long as it can be done without holding up the paper too long (and after Nature determines it's serious about publishing the work). However, it wouldn't be fair to do this if you didn't plan to publish the paper.

Nature: Is Nature's current review procedure adequate as it stands for refereeing of cloning papers?

Arnold Kriegstein: Specific confirmatory data should be required (see below), but it is not reasonable to expect that editors or reviewers will be able to ensure against concerted attempts at fraud. Papers with the potential to have a major impact on a field (not limited to stem cell papers) may be required to undergo a particularly rigorous level of review to guard against misrepresentation or misconduct. This would include, but not be limited to, particular responsibilities taken by the editors, including scrutiny of figures for manipulation and possible requests for submission of raw data which could be compared to final figures.

Nature: If not, what improvements are needed?

AK: Clearly DNA fingerprinting data will be required, imprinting data as well. It would be best for this to have been independently confirmed, if possible, with the data forwarded to reviewers along with the manuscript. The authors will also have to agree to outside testing, if requested, and should be required to store samples from both the somatic cell donor(s) and the oocyte donor(s).

Nature: Would the peer review process of cloning papers be improved were Nature to establish a checklist of minimal standard criteria for authors and referees to refer to during the peer review process?

AK: Yes, good idea.

Nature: If yes, what sort of experimental data should be considered and to what extent do the raw data need to be presented to reviewers, and to what extent should it be published?

AK: See above. It is always a good idea to present the data in as raw a form as possible, even if it does not look so good.

Nature: In 2006, Nature asked for independent verification of the Hwang paper showing that Snuppy is a cloned dog. Do you feel this kind of independent verification should be requested for all cloning papers? If yes, what should it entail?

AK: It is difficult to see how one can ask for independent verification for the authors, if not, who would do this and why?

A general point: I don’t think the guidance given by Nature regarding ethical considerations is specific enough and as with animal use, whose ethics and guidelines should we follow anyway? Different countries apply different standards. My view is that the major journals should establish a mutually agreed Charter that submitting authors have to comply with. Given the impact that papers in Nature and Science can have on careers in some countries (particularly some with questionable standards), this could be a valuable route to changing those standards.

Nature: Is Nature's current review procedure adequate as it stands for
refereeing of cloning papers?

Alan Colman: In terms of assessing the submitted information, I think the review procedure is as robust as it can be (but see below); a dedicated fraudster could deceive the most perceptive reviewer

Nature: If not, what improvements are needed?

AC: See below.

Nature: Would the peer review process of cloning papers be improved were Nature to establish a checklist of minimal standard criteria for authors and referees to refer to during the peer review process?

AC: I think coming out with a checklist that fits all would be nigh on impossible. I do think the journal should state that it reserves the right to demand independent verification of certain data, at the submitter’s expense. In view of this, the journal might wish to offer the recommendation (via an editorial as well as in the Guide) that if possible, scientists take steps to ensure that a deposit of, or access to, cellular starting material or extracted DNA is available for comparison with the experimentally derived cellular material. Ideally, the starting material would be available from an unimpeachable source eg in the Korean case, the living donor, or (less ideally) one or more depositories (these could be the labs of respected individuals that are not beneficiaries of the publication. In the case of Dolly (made in 1996), we were fortunate that the greater part of the mammary gland used to provide donor cells had been stored at another institution 100 miles away from Edinburgh since 1994. None of the scientists at that institute were involved in the Dolly research. When I organized for Alec Jeffreys to fingerprint Dolly, he was able to compare her DNA with that obtained from that stored material. We were lucky. The lesson is that scientists knowingly embarking on real landmark stuff should prepare right from the beginning. As we found with Dolly, even previously untarnished integrity counts for nothing.

Nature: If yes, what sort of experimental data should be considered and to what extent do the raw data need to be presented to reviewers, and to what extent should it be published?

AC: I am not sure that this would help. In the Korean case, there was mention post publication that DNA fingerprints seemed too identical; if this is a sign of cheating (I am not expert enough to know), presumably future fraudsters could doctor samples of identical DNA so that they gave very similar profiles sporting different amounts of noise (from addition of small amounts of third party DNA).

Nature: Last year, Nature asked for independent verification of the Hwang paper showing that Snuppy is a cloned dog. Do you feel this kind of independent verification should be requested for all cloning papers? If yes, what should it entail?

AC: Journals need to use discretion here. In landmark papers like Snuppy, the answer is yes. But the sanction should only be exercised once the paper is otherwise acceptable. This is because the process of verification is burdensome, quite expensive, and it would be unfair to put authors through this if the paper was going to be rejected on other grounds. The reviewers would be the best people to assess whether the project could be verified (for example, cloning using early primary embryonic cells might be unverifiable in this way because the source cells may all be used up during the experimental process). Further issues are: how do you ensure that kosher samples are available to the testing organization?; what is the nature of the organization; in the verification process, should some independent person orchestrate the sample collection and should the process be blinded with added, irrelevant samples, etc?. Clearly, a process of verification, which for some types of cloning could be absolutely definitive, would be impossible in many other scientific areas. Nevertheless, I think the general principle that independent verification should be sought where the stakes are very high in any area of scientific investigation, is a sound one so long as this process is practicable and rapid. The counter argument is usually that important findings are rapidly corroborated (or not) by others in the scientific community drawn to the area. Of course, this is true but it can take a lot of effort and involve the sacrifice of the sensitive resources like animals, to disprove certain high profile claims. In breaking scientific developments, a failure to reproduce an experiment is often thought by the experimenter to be due to a mistake on his or her part.

Nature: Aside from the 2004 and 2005 Science papers from Hwang's group, do other flawed cloning papers exist in the literature? If yes, please identify them and briefly highlight problems and how they could be rectified or prevented by the peer review process. Very difficult to say.

You know, we have sort of lost one leg of the scientific method and I blame it in part on you (Nature). If you were not so good at stirring up interest in your papers, our little community of cloners would be quietly sitting around waiting for somebody else to replicate Hwang's work. I don't advocate turning off your PR machine. It's good for science for the public to get charged up. But it sure would be nice that along with the dose of news, the persons on the street got a little help putting the observations in perspective.

Nature: Is Nature's current review procedure adequate as it stands for refereeing of cloning papers?

RW: Assuming your current procedures for refereeing cloning papers is the same as for other manuscripts, I would say your standards are adequate. They are higher than any other journals for which I have reviewed.

Nature: Would the peer review process of cloning papers beimproved were Nature to establish a checklist of minimal standard criteria for authors and referees to refer to during the peer review process?

RW: A checklist is not likely to be all that helpful. They probably will not fit individual cases all that well. After all, exceptions to rules are the norm for the stuff you publish. A checklist will likely be accepted by the authors without difficulty, but reviewers may take exception to them, or just ignore them. I suspect reviewers have probably increased the length of their mental checklist as a result of this episode.

Nature: If yes, what sort of experimental data should be considered and to what extent do the raw data need to be presented to reviewers, and to what extent should it be published?

RW: That is a good question. But I took it differently than you meant it. It crystallizes a point for me. If we can't believe in the integrity of scientists, we are going to have to invent one heck of a different system then we have now for getting new findings communicated. I could envision a process analogous to FDA's drug review. I pity the reviewer that has to start the process by looking at raw data. I've got enough trouble figuring out if they designed the experiments right. I don't what to have to check to see if they scanned the gel correctly. The review process is currently robust enough to pick up inconsistencies that might arise from sloppy execution. But I don't think the availability of more raw data is going to help the reviewer catch fraudulent practices. May be you need a staff trained in forensics to review the manuscripts.

Nature: Last year, Nature asked for independent verification of the Hwang paper showing that Snuppy is a cloned dog. Do you feel this kind of independent verification should be requested for all cloning papers? If yes, what should it entail?

RW: Well, that could be the beginning of an interesting experiment. But I would hate to see it go on too long with the consequence of increase the cost of my subscription to Nature. I predict that you will identify a little more sloppiness in the system then we might like to admit, but I don't think you are going to catch many folks trying to be fraudulent.

Nature: Aside from the 2004 and 2005 Science papers from Hwang's group, do other flawed cloning papers exist in the literature? If yes, please identify them and briefly highlight problems and how they could be rectified or prevented by the peer review process.

RW: I don't know of any such papers.

Nature’s standards have been recognized in the present controversy. You
asked for independent review of the genomic and mitochondrial DNA in the case of the report on dog cloning, according to media reports. This was an important insurance for the validity of reports on cloning. Perhaps this should be required for all scientific reports in cloning and related articles. The apparent reality of the dog clone is one of the few bright spots of the investigation.

Nature: If not, what improvements are needed?

AT: How to improve the certainty of evidence? I think independent examination of microsatellite nuclear DNA and d-loop mitochondrial markers of the donor and clone(s) or cloned ES cells. This will identify the origin and likely source of surrogate cytoplasm. This means that donor DNA/cell samples and cloned samples must be available to a nominated independent lab. There should be an independent letter of validity of samples provided as well. The review should include and
expert evaluation of genomic data provided (akin to independent
statistical evaluation in papers involving complicated statistical
analyses.

Nature: Would the peer review process of cloning papers be improved were Nature to establish a checklist of minimal standard criteria for authors and referees to refer to during the peer review process?

AT: I am not sure a check list will help, but it wouldn't hurt to provide this for guidance to ensure referees are aware of the potential pitfalls.

Nature: If yes, what sort of experimental data should be considered and to what extent do the raw data need to be presented to reviewers, and to what extent should it be published?

AT: I am not sure additional raw data is worthwhile to be published but reviewers should have access to as much as necessary for a complete
review. I think should be more than that usually provided at present by
Nature although I recognize reviewers can request this. The way data is
now presented in enhanced forms in figures for the primary journals, is
a concern because this not entirely accurate or at least not what we
actually see. I am even concerned when colors are altered (eg red for
GFP). Referees should be given primary photos and figs to help in
review- this is more general concern than the simple cloning type
papers.

Nature: In 2006, Nature asked for independent verification of the Hwang paper showing that Snuppy is a cloned dog. http://www.nature.com/nature/journal/v440/n7081/full/nature04685.html Do you feel this kind of independent verification should be requested for all cloning papers? If yes, what should it entail?

AT: See my answers to the first question.

Nature: Aside from the 2004 and 2005 Science papers from Hwang's group, do other flawed cloning papers exist in the literature? If yes, please identify them and briefly highlight problems and how they could be rectified or prevented by the peer review process.

I don't know of other flawed papers in cloning. I worry about closely
related claims such as evidence of transdifferentiation/pluripotentiality of mature cells, and claims of
germ cells in marrow that can spawn oocytes in ovaries within a few
days. These claims are probably more about misinterpretation than
fabrication, but a gray zone exists in this interface.

I think Nature should recognize the extraordinary does happen, and the
first report is worth recognizing but the second-independent confirmation is just as important. This means the confirmation paper
should be treated as a valuable addition to publish in Nature. I can
quote examples from my own experience. We had human ES cells in the
lab in 1998 when Jamie Thompson's paper was sent us for review. We held
off our paper until 2000 when we had directed differentiation into
neural cells because we believed the same data as Thompson's would be
rejected as not new. More recently we were able to confirm putative
oocytes could indeed be formed from ES cells - ie independent
confirmation of an earlier report that others couldn't replicate. Nature Methods wasn't interested, which is OK, and we sent it to Stem Cells.

All of this means that very new data tends to dominate and the
confirmation reports tend to languish. I wonder if you should create a
section for the "unexpected" which encourages confirmation, so the
community at large understands we haven't accepted the initial report as fact until independent confirmation is obtained.