Scientists are irked over a paper claiming, as The New York Times reported on Monday, that “DNA’s power to predict illness is limited.” “Yes,” geneticists have replied. “What else is new?”
Geneticists don’t dispute the idea that genes aren’t the only factor that determines whether we get sick; many of them agree with that point. The problem, geneticists say, is not that the study, published on 2 April in Science Translational Medicine, arrived at a false conclusion, but that it arrived at an old, familiar one via questionable methods and is now being portrayed by the media as a new discovery that undermines the value of genetics. Here are the main criticisms of the new study and the resulting press coverage:
1. This study critiques the power of genomic medicine but does not contain any genome data. The paper is titled, “The predictive power of personal genome sequencing,” but it doesn’t include any sequence data. Instead, the authors analysed data on how often twins developed the same diseases. Because twins have very similar genomes but don’t always develop similar ailments, the authors, led by Bert Vogelstein and Victor E. Velculescu of the Johns Hopkins Kimmel Cancer Center in Baltimore, Maryland, assumed that the frequency with which the twins got the same illnesses reflects the power of their underlying genome sequences to determine their health. This assumption is not true (see point 4), and isn’t a good basis on which to dismiss the value of genome sequencing in the absence of data from large genome-sequencing studies, which are just now getting underway.
“Let’s fast-forward a year or two, when we’ve sequenced a million or two million people in whole-genome sequencing studies,” says Eric Topol, a cardiologist at Scripps Health in La Jolla, California, and author of The Creative Destruction of Medicine: How The Digital Revolution Will Create Better Health Care. “Then let’s see whether or not the predictive capacity is limited, or limited for certain conditions but not others.”
2. This study is beating a dead horse. Many other studies have already found that genes alone don’t predict a person’s risk for developing most diseases very well. They’ve also specifically questioned the value of commercial genetic tests that promise to reveal users’ risk for various illnesses. The new study doesn’t acknowledge any of the previous studies that have already arrived at the same answer and have done a better job of it, geneticists say (see point 3).
3. The mathematical model used in the study is unrealistic. Geneticists have developed a slew of mathematical models that try to predict how likely a person is to develop various diseases. Scientists debate how well these models work, but the models are largely based on how diseases actually behave in the real world. The Vogelstein–Velculescu model is not, say statisticians.
Vogelstein, Velculescu and their colleagues first developed a model that poses a theoretical idea of how diseases might behave. They then tested their model against data from twin studies. The model divides the universe of human genomes into 20 groups, or “genometypes.” Each of the genometypes encodes a certain disease risk and occurs with a certain frequency, but the authors don’t know how often different genometypes carrying various disease risks occur. To figure this out, they ask which combinations of disease risk and genometype frequency are realistic by comparing them to what they actually see in twin studies.
The problem with this approach, statistical geneticists say, is that it uses flawed data to test unrealistic assumptions. Geneticists know how often certain genetic risk variants for various diseases occur in the general population, and how much risk each of these variants confers. The new model ignores this information, and instead allows diseases to behave in ways that differ from how they behave in real life. “The particular parameters in the model don’t really correspond to anything in terms of real world behaviour of genetic risk variants,” explains Luke Jostins, a statistical geneticist at Cambridge University, UK. “This divorces the model from population-genetic plausibility, making the results potentially meaningless.”
By ignoring information about how diseases act in the real world, the new model also allows the authors to sidestep some controversial unanswered questions, such as whether standard models overestimate the genetic contribution to disease in twin studies. That could be a nice feature of the model, geneticists say. But because of the limitations of twin data, combined with the authors’ flawed analysis of these data (see point 4), there’s nothing in the paper to ground the new model in reality. If this were the first-ever paper to try to define the limits of genetic-disease prediction, it wouldn’t be convincing, says Jostins, who also blogs at Genomes Unzipped. “It’s very hard to interpret this model,” he says.
4. The study doesn’t correct for errors that can affect twin studies. The study assumes that genetics is the sole factor that determines whether two twins develop the same disease. But twins also grow up in a common environment, and the study doesn’t account for this, as the authors admit.
It’s also rare for both members of a twin pair to develop the same disease. So even a study such as this, which combines data from many different twin studies, suffers from a relatively small overall sample size of affected twins. That lowers the statistical reliability of its findings and introduces unpredictable errors into the study, Jostins says. Again, there are ways to account for for these errors, but this study doesn’t try to do that.
5. The media coverage of the study could weaken support for genetic research. Geneticists have lobbed some pretty heavy artillery at the Science Translational Medicine study, even though it claims to affirm what they already know. That’s because the new study has received more press coverage than your run-of-the-mill statistical genetics paper, and geneticists are concerned that the coverage has overblown the study’s conclusions in ways that could harm public support for science. “I don’t see the harm in telling the public yet again that there is no such thing as genetic determinism,” says Leonid Kruglyak a geneticist at Princeton University in New Jersey. “But I worry about the message being distorted to mean that genes have no value, or that genetic research is not worthwhile.”
Follow Erika on Twitter at @Erika_Check.
Report this comment
Valid comments, I agree with them!
When next generation DNA sequencing will be applied to to various ethnic DNA collections, the signal to noise ratio will increase as well as the predictive value of whole genome DNA sequencing!
Emanuel Yakobson, Ph.D.
Professor of Ethnic and Personal Genomics, University of Latvia, Riga
Report this comment
Sadly, I also saw on a media report a comment from a woman with a serious cardiac mutation—known in her family, and she has already tested positive. She seemed to think that discounting and undermining genetic testing was a huge relief. That could be an unfortunate conclusion if she blows off her test result now.
Report this comment
@Mary Mangan
That is terrifying. Would you be able to dig out where you saw that?
Report this comment
Yeah. It’s on the CNN article called A warning against genetic testing in case the link doesn’t work after I post it.
Here’s the comment that disturbed me: “After my brother went into cardiac arrest brought on by a hereditary disease, I was tested for the gene, hoping and praying I didn’t have it. Unfortunately I do and I’ve been living with this fear I’m going to have the same problems. This article is good news to me. It takes some of the fear of the unknown away.”
Link to CNN piece
Report this comment
@Mary Mangan
That is terrifying. Would you be able to dig out where you saw that?
Report this comment
So who is being irresponsible here? Are we, as scientists, considering the impact of over-interpretation of our work? Is it the job of the reviewers to treat all submissions as potential dynamite? What about the editorial inertia? Is it the place of the scientific journalist to dig deeper? What about the more popular press that picks up a science sound byte or two? It seems to me that while we all share some responsibility, there is a hierarchy and it starts with the originating scientist(s). There is some level of conflict of interest at each stage but the source has most to gain (and lose) and must be transparent. But the reviewers must also recognize that authors have to “sell” their work, especially if it is controversial or anti-dogma. How much role did the editors play? For all we know, the reviewers expressed major concerns and were overruled (transparency efforts at EMBO include publishing the review history and documentation – but only for published articles). It’s certainly a good sign that some scientific journalists take to time to parse the complexities of experimental design and data analysis but they should not be expected to “police” research. Indeed, as soon as they do, it is a red flag that our vetting system has failed.
Report this comment
One of the newer fields in genetic study, epigenetics, is attempting to bridge genetic causes of disease to environmental conditions. This would include lifestyle. One example would relate to the twin model where one twin develops disease and the other does not. Environmental conditions such as smoking, exposure to sunlight cause damage to the epigenome, cell factors that control gene expression. Therefore, having the same genetic makeup does not necessarily lead to development of the disease.