Skip to Content

Trouble with CRISPR? Maybe, But Maybe Not

Now, here is a paper that is causing a stir. Published in Nature Methods, researchers from Stanford, the University of Iowa, and Colombia are reporting that CRISPR treatment in mice introduces far more mutations than thought. The news sent the stocks of companies like Editas and CRISPR Therapeutics dropping yesterday, and has led to consternation around the biomedical research field.

First, what this paper reports: after screening four small guide RNAs (sgRNAs), they picked the one with the most activity to go into mice. The two treated mice (F03 and F05) were whole-genome-sequenced at an average coverage of 50x (controls at 30x), and the two sets were compared for insertions/deletions (indels) and single-nucleotide variations (SNVs). And here’s what showed up:

All pipelines showed that F03 harbored 164 indels and 1,736 SNVs (63 and 885 of these, respectively, associated with known genes). F05 harbored 128 indels and 1,696 SNVs (51 and 865 of these, respectively, associated with known genes). . .The same 117 indels and 1,397 SNVs were detected in both of the CRISPR-treated mice, which indicated nonrandom targeting. . .The mutation rate detected in CRISPR- treated mice was substantially higher than that generated by spontaneous germline mutations (3 to 4 indels and 90 to 100 SNVs, de novo, per generation).

Oh dear. If that holds up, that’s clearly going to be a major difficulty in bringing CRISPR-based therapeutics forward, at least with the current state of the art. Just as worrisome, if not more, is that fact that software predictions of the fifty most likely off-target sites of action did not match any of the variants that were actually seen. So it would appear that we have no idea of what’s going on here. As the paper says, with great restraint, “The unpredictable generation of these variants is of concern.”

Now we get to the arguing, though, because the big question is whether these results are correct. They do not match up well with what’s already in the literature on the subject, that’s for sure. On Twitter, Nicolas Bray brought up one of the concerns. His question is a simple one, but it needs to be answered: the two treated mice were siblings, while the control was (apparently) more distantly related. How many of these variations, then, can be ascribed to what the mice started out with? The control animal was from the same inbred strain, but still – the number of variations seen in this paper is way off what others have reported. Sam Sternberg noted that the paper (and its supplementary information) is not very clear about the relationship between the mice, and also suspects that many of these mutations are from the founders and not the CRISPR treatment per se. Meanwhile, Gaetan Burgio pointed out that the experimental details say that “Briefly, an sgRNA- expressing plasmid had been coinjected, into FVB/NJ zygotes, with the single-stranded oligodeoxynucleotide (ssODN) donor template and Cas9 protein to generate mosaic F0 founders“. Plasmids themselves, he notes, are known to cause mutations, since they have much longer half-lives than RNA or protein, and he says his own lab’s work with direct injection of sgRNA and Cas9 protein showed far fewer mutations, in keeping with the rest of the literature. Another potential problem has been brought up by many observers: this study has an n of 2, with one control animal. That’s a pretty thin platform on which to build the “CRISPR Is Doomed!” monument.

So there are a number of reasons to wonder if these results are real. If they are, other questions arise about the newer Cas9 variant enzymes and various sgRNA designs, but I’m not even going to think about those, to be honest, until this result has been replicated and given a thorough going-over. The implications are too big to start running around in circles just yet. There will be time for that, if needed. . .

Update, in the interests of full disclosure: after publishing this post and looking at the moves in the stock prices of both EDIT and CRSP yesterday and today, I found my own doubts convincing enough to have bought some of both as a short-term trade. I have no idea what their long-term prospects are, but their sudden drops due to this news may not be justified. On the rare occasions that I’ve taken a position in any sort of individual biopharma stock, I’ve noted it on the blog, and will continue to do so. Now you all can watch me lose money in real time. . .

47 comments on “Trouble with CRISPR? Maybe, But Maybe Not”

  1. NMR4life says:

    Interested to hear from those in the genome editing field. I get the feeling that CRISPR has been over-hyped relative to other techniques, likely due to the pedigree/resources of the institutions looking to profit off of this.

    1. Derek Lowe says:

      I think it’s the real deal, actually – its specificity and ease of use really do stand out. But this latest stuff has to be ironed out, and quickly. . .

      1. Bryan says:

        Isn’t that what people were thinking about siRNA a decade ago?

        1. Sam says:

          Exactly!

      2. Mol Biologist says:

        CRISPR is a bacterial immune system which design by nature to eliminate all viruses and phages as enemies. The specificity is not highly required in this defense since it is targeting repetitive sequences or palindromes. Human and mouse genomes are intruded with mobile genetic elements both “hot” and not active which also contain palindromes. Even if you think your “RNAs” guiding is specific it is not. So, it is as a shooting by grape shots you are always getting multiple hits. But as buisness model it is very profitable since you must “cook” different “RNAs” every time. No matter where it’s done in mice or human embryos, you’ll get another 1000 hits. I would rather stay away from this type of treatment.
        https://www.nytimes.com/2015/04/24/health/chinese-scientists-edit-genes-of-human-embryos-raising-concerns.html?_r=0

  2. There have also been a number of comments on Twitter from the genome variant calling experts questioning the mutation calling in the paper. Done improperly, it is very easy to have many false positives.

    Because of the paywall, I haven’t seen the experimental details. Did they use the most recent engineered Cas9 which have higher specificity?

    There’s a very real danger that these results are being overhyped despite using last-generation Cas9, improper experimental technique and improper data analysis. As someone else has pointed out on Twitter, if this had been a preprint these issues could have been hashed out before the imprimatur of a Nature journal had been stamped on it.

    1. Derek Lowe says:

      I think that this was done with “classic” Cas9 – they mention near the end that similar experiments need to be done with the higher-fidelity versions.

      1. Yup says:

        So they ran a poorly controlled experiment with obsolete technology? How did this get published at all?

  3. Thoryke says:

    I don’t think there’s anything especially outrageous about their final statement: “Future studies employing new CRISPR methods and reagents should consider using WGS to determine the presence of off-target mutations in vivo.”

  4. johnnyboy says:

    This isn’t my field, and admittedly this probably represents a fair amount of work. But still… two mice ? In a Nature journal ? With results broadly contradictory to what is already published ? Who the hell publishes anything with a n of 2 these days ? Do reviewers just let anything through if it has CRISPR in the title ?

    1. matt says:

      The thing that bugs me most about this is that it’s not even a peer-reviewed paper, it’s just a letter to the editor! For something this scientifically high-profile that’s also almost certain to move the stock prices of multiple companies, it seems borderline irresponsible to publish it without peer review. The press is hardly going to make the distinction for us — even Derek here refers to it simply as “a paper published in Nature Methods,” which anyone might reasonably assume means it was peer reviewed. It was not.

  5. mb says:

    I wonder why they did sequence the mice pre and post treatment. Case/control does not seem the best method here. Although you could include it with pre and post treatment.

    1. Barry says:

      They injected the CAS-9 and the plasmid for the small guide into zygotes. There was no opportunity to screen the “mouse” before.

      1. mb says:

        embryo splitting could have been done, especially with only 2

        1. Barry says:

          they injected the zygote. You can’t split a single cell.

          1. mb says:

            wait till it is zygote, it is only 1 cell division, divide, inject 1

        2. bazilione says:

          The reagents (Cas9 and guideRNA) would be in the same cell (zygote) – the cells after splitting would not be a proper control.
          They should have chosen the parent mice better.

          1. steve says:

            mb is right – the right way to have done the experiment would have been to split at the two-cell stage, sequence one cell and inject the other.

    2. Barry says:

      Because as the paper states “an sgRNA- expressing plasmid had been coinjected, into FVB/NJ zygotes” there could be no genome sequencing before the experiment because there was no mouse before the experiment. That would require a non-destructive sequencing techology.

  6. mallam says:

    I’m surprised the stocks didn’t crash more. Looking at their historical prices and until better data, the upside will be limited.

  7. Ali says:

    I would be very careful to draw too much from these results with n=2. It is a real shame this article is taken to such high level of publicity.

    One can argue the data is fundamentally flawed and this would be my conclusion until these data are validated independently with a larger group of mice. Pre-sequencing untreated mice is a must to confidently call a mutation resulting from Cas9 treatment. I am highly sceptical of the data knowing the two treated mice were siblings. My assumption is that most of the SNVs are NOT related to Cas9 at all. This together with the fact that most of the SNVs are identical in the two treated mice where the target site is independent of the target site in vivo and none of the top 50 predicted off-target were effected suggests these data are highly questionable.

    Let’s not panic and wait for more independent data.

  8. Calvin says:

    Of course the CRISPR folks want to explain this away, but I think it’s been known for a long time that CRISP isn’t quite as specific as the sales hype would suggest. This has been extensively discussed/debated in the microbiology field. There’s little doubt there are off-target issues, but whether these result in a bad phenotype is less clear. That’s not to say that CRISPR isn’t the real deal, very useful and could lead to some amazing clinical results, but it’s not a silver bullet; there will be unexpected issues/problems. In our industry we always think the latest technology is cool and shiny, but as with everything else, as we know more, they become more battle worn and less shiny as we acquire more data. So I’m willing to bet that this publication is a decent reflection of the fact that CRISPR isn’t as precise (which may be fixable) as we might have hoped. It might not be this bad, but it might…..

  9. bhip says:

    As some comments have noted above: 1- They must have tested before & after transfection with treatment & control vectors ???!(if not, it shouldn’t be published anywhere); 2- Sample size is ridiculous and; 3- Most importantly, never assume (software predictions) when you can test (whole-genome-sequence- pretty doable these days).
    I would suspect/hope that upon clinical testing (or preferably in tox testing), genomes would have to be sequenced before/after treatment.

  10. CRISPR Little was right says:

    Plasmid vectors causing unplanned mutations? Are we doomed as a species and will CRISPR escape into the environment and damage our genome randomly wiping out our species and all known life forms? Has the genetics and its interaction with the environment been studied by science enough as all the GMOs are generated, reproducing, and passing random plasmids?

    Stay tuned America and all those possessing DNA as CRISPR undergoes safety scrutiny for what seems to be the genetic version of Thalidomide in the 60s!

    1. DavidDeesIRL says:

      Not convincing, not even a single mention of chemtrails or fluoride. C for effort.

      1. steve says:

        My guess is that the results are all due to peroxynitrite.

  11. Ashutosh Jogalekar says:

    For me, problems with CRISPR have always seemed similar to the two basic problems with drugs: efficacy (does CRISPR edit all possible genes) and toxicity (does it edit ‘off-target’ genes). Studies like this one seem to indicate that just like with drugs, we are stuck with these two problems for CRISPR for some time.

  12. MrRogers says:

    I saw this and thought it must be a troll. We work closely with our core and have generated multiple new alleles (us >10, them >50) over the last couple of years and neither we nor they have seen anything like this. (Yes, we sent the first few founders out for WGS). However, we have only done mRNA or RNP injections. (RNP is more specific and more effective in our hands). It makes me wonder whether this group wound up with the Cas9-expressing plasmid integrated into the genome.

    1. Mark Thorson says:

      Did they buy any reagents from Santa Cruz Biotechnology?

  13. Trumpster says:

    Dont worry, we’re going to generate such great fidelity Cas9 that your head will spin! The new Cas9 is going to so good and will come out so fast that you’d so amazed. Oh boy are you going to be so amazed. You’ll be so amazed that this paper will get knocked the hell out. It’s time to make genome editing great again, make genome editing safe again, and make genome editing work again. Oh we’re going to put all those enzymes to work, believe me! I alone can make this work. Genome editing is going to be really really revolutionary that you can even imagine. Genome editing will work, will work and will work! You better believe it. Together, let’s make genome editing work again!

    1. Mol Biologist says:

      Love it. Very dramatic. No doubt a lot of people will try to repeat it again and again and again.

      1. Mol Biologist says:

        I will not be surprised if the next project will be called Innovative CRISPR/Cas systems for cancer treatment. And it sounds again very profitable. May be or Maybe not.

  14. Scott says:

    Man, do I need to get back into the Rules of Statistics?
    1) Compare like with like, anything else is a waste of time
    2) Minimum of n=30 or don’t even bother reporting
    3) n=1000 is better than n=30. Greater than n=1000 is not necessarily required, as sample size to improve predictive accuracy goes up logarithmically.

    Any alleged “research” that violates Rules #1 or #2 gets red-carded and thrown back for rework with extreme prejudice.

    This “study” violates Rule #2 and should have been rejected at the design stage!

    1. pete says:

      exceptions to your rule 2 (i.e., that n should =30, or greater)
      The electric stove test:
      [Hypothesis: I will get burned if I touch the red burner, but not if I touch the black burner]
      Here, an N of 2 should be quite sufficient

      1. Scott says:

        I’ve seen lots of “black” (non-glowing) burners hot enough to blister skin instantly, so there’s a good chance of false positives (this black burner just burned me!) there. But boy does it suck to be the tester for that hypothesis!

        So I’m going to stand by my Second Rule of Statistics, even if it’s my hand testing the burners.

    2. Diver Dude says:

      “If you need statistics, you did the wrong experiment”
      Ernest Rutherford

  15. Li Zhi says:

    The real question is why Derek bought the stock and not an option on the stock.

  16. chemkev says:

    Here is another paper reaching similar conclusions:

    https://www.nature.com/articles/ncomms15464

  17. Chrispy says:

    I have to applaud the authors for at least pointing out a potentially serious problem with a technology they are heavily invested in. I know of many labs that would have swept this under the rug, sad to say. Yes, the numbers are small, but isn’t WGS a relatively expensive and time consuming effort even now? Perhaps it is rather a lot to expect this to be done in lots of mice, though I would have liked to see more, too. Anyone know what the going rate for WGS is?

    It is interesting that many of the mutations are shared between the two treated mice. This suggests that it might be a solvable problem (or confounding at the early stages of setup). The fact that the current predictive algorithms appear to be so bad suggests we have a long way to go, though!

    1. Ali says:

      Indeed it is very interesting many of the mutations are shared between the two treated mice. The two treated mice are also siblings. With the control mice being distantly related, there is nothing to say that most, if not all of the SNVs have in fact nothing to do with Cas9 and these SNVs are inherited from their parents.

      As I mentioned above, let’s not panic and wait for more rigorous and independent data. By the way, I have no incentive to defend CRISPR/Cas9 as all I want to see is proper rigorous scientific data.

  18. Spike says:

    I think that this must be the first time that I have seen a publication where the number of authors (6) and affiliations (7) outnumbered the number of animals used (3). I guess that they will get some kudos in terms of the 3Rs

    1. Pennpenn says:

      A case to suggest the mice are experimenting on the authors/affiliates?

  19. loupgarous says:

    An n of 2 (even with Fisher’s Exact Test, which famously gave p-values of 0.14 with an n of 8 which were borne out by “experimental results” – the tea tasting experiment) is guaranteed to create p-values far above the old significance cut-off of p=0.05. much less the new standard of p=0.005.

    The other problems with the study (siblings used for the “effects” group when one of them clearly ought to have been a control animal) combined with ridiculously low sample size to make this an anecdote, not a study of significant results. As such, it ought to spur workers to repeat the study with a larger sample and more responsibly chosen test and control animal cohorts.

  20. Guido Gambardella says:

    Excuse me, I have a better understanding of genomics than 99% of laymen, i.e one thousandth of the author of this article and the people that commented it.
    But I also know a couple of things about economics, and I do believe that the effects of that letter have been beneficial, since it punctured the speculative bubble that boosted way too much CRSP and EDIT.
    It reminds me of the fake profit warnings Nokia (an honest company) issued in the early 2000’s, and I would not be surprised if the letter was written in accordance with CRSP and EDIT.

  21. StockTrader says:

    Not a bad trade Derek! 20% in a couple of months!

    This is how it’s done folks – Trade what you know!

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.