Nature Methods | Methagora

Minimizing the risk of reporting false positives in large-scale RNAi screens

A group of investigators from various institutions have written a Commentary in which they articulate their experience about caveats of large-scale RNAi screens. These authors propose that the use of standard controls should be adopted across the community of RNAi users to minimize the risk of reporting false positives among screen results.

Far from trying to dictate rules, this Commentary seeks to open a community-wide discussion leading to a consensus of best practices that can realistically be implemented across all experimental settings. Therefore, we welcome your reactions and hope for a constructive dialog for the benefit of all. Please post your comments below.

The urgent need for such debate has been highlighted in two recent papers by Ma et al. in Nature and Kulkarni et al. in Nature Methods. These two studies demonstrate that RNAi screens in the Drosophila melanogaster system—in which long dsRNAs are used as RNAi triggers—are affected by off-target effects to a much wider extent than previously thought.

As far as the use of RNAi in mammalian systems is concerned, off-target effects have been a constant preoccupation. Nature Methods’ recent focus, RNA interference – A user’s guide, contains practical discussion of issues revolving around the use of RNAi as a research tool in mammalian systems, and includes notably a Perspective by Yi Pei and Tom Tuschl on effective siRNA design and a Perspective by Bryan Cullen on experimental design to enhance and confirm specificity. The latter is, however, focused on experimental situations in which one aims at knocking down the expression of a single, specific gene of interest in mammalian cells.

For those involved in genome-wide screens in various model organisms, like the Commentary’s authors, the recommended controls may be technically difficult to implement—hence, the need for a discussion of appropriate measures to strike the right balance between practical feasibility and accuracy of published screen results.

Comments

  1. Mika Rämet said:

    I am pleased that Echeverri et al. brought up an important issue of the quality of RNAi screens in the October issue of Nature Methods. Ever since we demonstrated the power of in vitro RNAi screening in Drosophila I have followed the use of this technique with great interest. For my disappointment I have had concerns about the quality of many of the published screens. For example, screens that have aimed to identify genes involved in the same process have yielded results with poor overlap. In addition, we have had problems repeating the results of several published RNAi screens. Therefore I would urge following nine steps to be followed in order to minimize the risk of reporting false positives in large-scale RNAi screens in Drosophila.

    1: Careful assay set-up. In my opinion, this is the most critical part of the screen and cannot be emphasized enough. Several questions need to be solved beforehand. Is our assay reproducible and quantitative enough? Do our positive (and negative) controls give the expected results? And perhaps most importantly, is it really possible to carry out our assay reliably in the 384-well format as it is done commonly nowadays? None of our own published or unpublished screens could have been carried out appropriately in such a way.

    2: Biologically meaningful cut offs. Every RNAi affects SOMETHING in the cells. Every time your silence something affecting – let’s say – transcription or translation you will have a lot of secondary effects that are likely to affect your assay. This may have a modest (but statistically significant) effect on the phenotype you are monitoring. However, it has nothing to do with the process itself you are interested in. Therefore I would recommend setting a biologically meaningful cut off based on the results obtained from dsRNAs targeting the known components of the process. This will drastically cut the length of the hit list.

    3: Careful assessment of the viability of the cells. If the dsRNA treatment affects the general well being of the cells it will likely also affect the selected read-out. Intolerably many dsRNA treatments that are reported to cause a specific phenotype have simply caused decreased viability in our hands. There are many ways to assess the cell viability and at least two independent methods should be used.

    4: Confirmation of the dsRNA. Once something biologically relevant is detected, one needs to confirm that the phenotype is really caused by silencing of the expected gene. Regardless of the source of the template for dsRNA synthesis, there will be some wrong dsRNAs in a large-scale dsRNA library. The production of the templates is usually done with PCR amplification and as we know, occasionally PCR reaction produces more than one product and sometimes even a wrong one. Therefore the obtained phenotype may be caused by a contaminant in the dsRNA preparation. Therefore investigators should

    5: repeat the positive findings with at least one other dsRNA targeting the same gene. As proposed by Echeverri et al. positive results need to be confirmed by repeating the results with at least one independent dsRNA targeting the same gene. These five steps mentiones above should shorten the hit lists so that the following four steps are feasible.

    6: To test whether RNAis against the closest homologues give similar phenotypes. This will partially rule out the possibility that observed phenotypes are caused by off-target effects silencing the homologues (effectiveness of these RNAis on mRNA levels should also be tested appropriately). Of course, homologues may also be involved in the investigated phenomenon. Of note, we have ourselves assessed the specificity of RNAi in Drosophila cells using microarray technique and found it to be totally specific in that particular case.

    7: Secondary and tertiary assays with different read-outs. Regardless of the read-out there are likely to be some positive results that are artifacts of the assay. Therefore it is essential – as also proposed by Echeverri et al. – that secondary (and tertiary) assay(s) should be used to confirm the results from the primary screen. These assays should be as different from the original one as possible. For example, if reporter assay is used in the primary screen, secondary assay might measure the amount of endogenous mRNA.

    8: In vivo validation. When Drosophila is used one should consider confirming at least selected positive(s) in vivo. After all, it is primarily in vivo model and this benefit should be fully utilized.

    9: Relevance in other model systems. Finally, to conduct a really ground-breaking study, selected positive finding(s) should be confirmed in an appropriate other model animal.

    Yes, this is an awful lot of work but after all this, obtained results will be solid and will provide appropriate basis for future research.

    Mika Ramet

    Institute of Medical Technology, University of Tampere, and Department of Pediatrics, Tampere University Hospital, Tampere, Finland.

  2. Heather True said:

    For those of us that have had the ability to do genome-wide screens in other systems for some time now, we all realize that any screen is only as good as the secondary tests you can design, and then they must be performed to rule out the false positives. This RNAi system is no different and it is probably not really feasible to simply use a few standard or universal controls to validate the screening procedure in general. Performing a screen knowing that many of the hits will not pan out is perfectly acceptable as long as the appropriate follow-up work is done to verify good candidates and then apply that knowledge in a meaningful way.

    Report this comment Cancel report
    Your details

    Please confirm the words below

    In order to reduce spamming, this process ensures you are a real person and not an automated program.

  3. Charles Towne said:

    I agree that this is an issue affecting a very large base of people studying mammalian cell systems (including myself). And although the scale is different for high throughput screening studies, this same issue has been raised a number of times with groups studying only a handful of proteins (genes). I agree with an earlier poster that results should be repeated with more than one siRNA/shRNA/etc. However, the assay design is far more important.

    I recently completed some experiments looking at 1) the affect of different control siRNA’s on cell viability and cell signalling, and 2) the ability to “rescue” an observed knock-down phenotype by transfecting the cells with a cDNA encoding the knocked-down gene that was not susceptible to the siRNA. I believe that this, (next to knock-out animal models) is the most solid address of confirmation of a phenotype. Giving credence to a knockdown phenotype because it was expected can introduce bias and ethical questionability, and should not be used as a reason to “believe the data”.

    Past studies have shown that siRNA’s targeting the untranslated region (UTR) of mRNAs can be as effective at knocking down mRNA levels as those that target the open reading frame. Therefore, I used siRNAs (notice the plural) targeting the UTR of one gene, then demonstrated the effective knockdown of the gene by real-time PCR (over 80% reduced)… A person might argue that the phenotype I observed was expected, but the phenotype did not go away when the same siRNA-transfected cells were transfected with a plasmid containing just the open reading frame of the gene (eliminating the siRNA target-sequence). Expression of the protein by siRNA/plasmid-transfected cells was confirmed by western blot… All this was done comparing the cells transfected (or not transfected) with control siRNAs (again the plural)

    The same experiments were repeated by targeting a second unrelated gene (but still within my field of study)

    Conclusion: that the SEQUENCE of the siRNAs and not the transfection protocol is the culprit for determining whether there will be off-target effects… The only way to definitively show that a knocked-down gene is responsible for a given phenotype is by trying to rescue the phenotype.

    Therefore, DESIGN the knockdown assay with a future rescue experiment in mind.