# Worse Than Useless

Time for another look at AbbVie’s work on Rova-T (an antibody-drug conjugate targeting the tumor antigen DLL3), and for some hard thoughts about what drug development is really like. The last time I wrote about this program, things didn’t look good. Now they look even worse. A Phase III trial of the drug has been halted because the treatment group is showing higher mortality than the standard-of-care control group.

AbbVie bought Stemcentrx for nearly six billion dollars (cash, not biobucks!) in a deal that I characterized as “either very smart or very stupid” in order to get their hands on this therapy and its follow-ups, and right now it looks like that entire investment is in danger of being a lost cause that will never be recovered. But let’s step into the time machine. We don’t have to charge up the batteries very much, because we’re just going to go back to 2015. That’s when the first clinical results for Rova-T were reported. The paper called it a “promising first-in-class” treatment. Now we move up to 2016, and trial results in small-cell lung cancer. That was not as impressive a showing, but still characterized as “encouraging”, with “manageable toxicity”. If you go back to March of this year, the further results made it look like the compound was, at best, only about as good as the standard of care, and a vast amount of money was trimmed off AbbVie’s market capitalization. In June the company presented an even more thorough look at the data, which made the entire program look like it wasn’t even as good as currently available therapies, and thus nearly pointless.

And now it’s gone past pointless to dangerous. This drug has, within the space of three years, gone from looking as if it would save lives to being statistically proven to accelerate death. And here’s the key thing to realize: this is not some weird outlier. Sure, AbbVie paid a lot of money for this, and they probably should have waiting to see some more clinical data before jumping in. And sure, Stemcentrx has the whole Peter-Thiel-Silicon-Valley history behind it. And it’s also tied into the whole complicated story of cancer stem cells. But those (interesting as they are by themselves) are honestly side issues, and they don’t really have that much to do with the clinical success of Rova-T. This story is not worth paying attention to because it’s so different – it’s worth paying attention to because it happens all the damn time.

It really does. It’s a constant feature of drug development that things look much better in the earlier trials than the later ones. That’s because the early trials involve carefully selected patients, but it’s especially because the early trials are very small. Unless you’re dealing with a massive, epochal throw-away-your-crutches effect size, human biology is complex enough to make small trials only very crude indicators. Signs of success are only that – signs. You need to go into more patients, of more varied types and in more varied situations, more times. Every time. There is no substitute for large, well-controlled, well-designed human trials. No one likes to hear that, because everyone’s heard it before, and because it’s not exciting and new and holds out no promise for huge accelerated money-saving breakthroughs. But it’s true.

If you’re a biopharma investor, you need to constantly remind yourself of that. If you’re a reporter, you need to do the same. And if you’re a potential patient waiting on such a new therapy, it’s very hard news to be reminded of this, but reality is preferable and this is reality. In the clinic, most drugs fail. They don’t all fail in the brutal Greek-tragedy way that Rova-T is failing, but most drugs fail.

And finally, if you’re an advocate for getting the FDA out of the way by whatever way you want to call it, because you’re convinced that it’s some sort of big slow roadblock to live-saving drugs reaching needy patients, I have something to say to you: Rova-T. It was new, it was promising, it was exciting, it was first-in-class, its toxicity was manageable, its early results were compelling, and a big drug company thought it was worth six billion dollars in cash up front. You would have given it to all those needy patients right then, back in 2016. Wouldn’t you? And killed them faster.

## 52 comments on “Worse Than Useless”

1. CuriousChemist says:

I am reading your blog carefully since few years now and I am always so surprised with this kind of stories.

We heard already few examples about this problem and the fact that clinical trial on a small population is most of the time not enough to draw big conclusions. But how can big pharma and all the clever people involved in such decisions ignore that and go all in with 6 billions cash in front and hope for the best? Are they so scared that another pharma with jump in and that they might not make enough cash?

I am really wondering what these meetings look like : Ok this looks good and we might make tons of money with it, it’s a poor set of data but lets go all in and pay 6 billions?

Maybe I am just a naive chemist but this is not the first time I am wondering…

1. John Wayne says:

There isn’t an obvious better way to do it.

In fact, this was supposed to be the better way to do it and here we are.

2. johnnyboy says:

Because in a big pharma, the ‘clever people’ are not the ones making the acquisition decisions. Often they are the ones advising the C-suite to wait before jumping, but to no avail.

1. Anon2 says:

+1

2. yf says:

Actually, they are so clever in sustaining their paycheck that they lost their integrities ( both personal and scientific).

2. Bagger Vance says:

That last paragraph would be pretty damning if you had any, whatchamacall, data saying that in no case ever (eg, “successful drugs”) the bureaucratic foot-dragging didn’t lead to denying people access to effective new therapies, which therefore also “killed them faster.”

Surely someone has done the work to see what the tradeoffs were, in life expediencies lost or gained, right?

1. Derek Lowe says:

That’s a rational argument, but it lands into “impossible to prove a negative” territory. What I can say, though, is that with clinical failure rates of about 90%, that the odds are not on the side of saving lives earlier by opening access faster.

2. Some idiot says:

Er, that’s not what he is saying. What he is saying to proponents of “right to try” is that this is a perfect example of what can happen when something looks good in early stages, but bombs out later.

What you are asking is whether or not this case is in the majority or not. I don’t have the data to point one way or the other. But considering that 90% of drugs going into the clinic never reach the market, I would say that this suggest this sort of case is more “rule” than “exception” (although probably a bit on the extreme side…).

1. Bagger Vance says:

No, Derek’s response was more what i had in mind. But i also doubt most drugs (success @10-1 odds) are really that dangerous after stage 1 or 2 testing–9/10 aren’t likely to be potentially deadly.

The potential responses are of course positive, negative and neutral–the drug does nothing beneficial but doesn’t have appreciable toxicity either. Derek comes out quite strongly against placebos IIRC but if 1/10 work well, 1/10 work against you and 8/10 don’t do much of anything, you may have as strong a case for accelerating things as decelerating them.

The need to eliminate the negative would be balanced against the potential benefits in a totally rational regulatory environment. How far you think the current system deviates from that is something that available data may support or not.

1. Chronically ill person says:

I was involved in a phase II study a while back and was in the open-label part of the study. It was stopped early because of lack of evidence of efficacy in meeting the primary end-point.

I questioned why they would stop the trial early if it didn’t now any toxicity and had the chance of helping.

Exact text and response:
Me: “If there was no safety indication, why terminate the trial
early, surely continuing the study until at least 12 months
would give you a chance to see if there was any positive
benefit for the patients considering the expense and labor in
enrolling and establishing the trial, especially considering
anecdotal evidence from some of the patients that they felt
more stable on the drug?

Response: “The ___ indicated that the recommendation to terminate
Study ___ was not based on the detection of a safety signal.
However every treatment has the potential for adverse drug
effects. If a therapy is shown to have no benefit in a
population, then continuing treatment offers no benefit,
only risk.”

And this is actually a good answer. It indicates that ethics are important. In the absence of evidence that your drug helps, all you are doing is exposing people to risk. And that’s why it can’t be the way you are arguing for, even though I also wanted that.

2. T says:

Well those 8/10 may not do much of anything, but they will almost certainly have side effects (even if not dangerous ones), and more importantly, if you are taking one of those based on the hype, the chances are it means that you are taking it instead of something that could actually help (e.g. the standard treatment it was hoped the new one would be better than, or another new one that has been shown to work, but not at a level that matches the hype for the untested one), in which case your choice of that useless drug is indeed doing you serious harm, even if its not acutely toxic.
I also wonder how all these patients taking whichever unproven drug candidate has the best hype would affect recruiting patients for Stage III clinical trails. This approach could delay the proper testing of drug candidates to find out whether they really work, which really would delay the availability to patients of effective new treatments (because they won’t know which one of the ten on the market is the one that could actually save them).

3. Ursa Major says:

“The potential responses are of course positive, negative and neutral”

There is also a fourth potential response: positive but not worth pursuing for non-medical reasons, e.g. it helps patients but not as well as a competitor’s drug so economically (and ethically) not worth continuing to develop. I imagine a lot of hope and cash is wasted on this category.

1. RTW says:

You are correct sir – Particularly in areas where there is ample needs met. Sometimes the drug as its going through the clinic it is also doing so behind competitors drugs having similar mode of action. Perhaps because of a late start, inability to enroll enough patients of other factors even if your drug passed muster it would be too late to market to re-coop costs in competition with its peers. Or it could be an acquisition resulted in a not invented here situation and it was dropped in favor of less further along drugs that were invented here….

3. MrXYZ says:

Here is one way to possibly think about this. During early (small) clinical trials, a bad result could be due to the drug not working or due to the statistics of small patient number. In either case, I would guess that business considerations, rather than the FDA, are more likely to kill a program at this stage. Why move something forward if it doesn’t work?

Conversely, the early clinical trials could give great results, once again due to either chance or a truly meaningful result. Going to Phase III, you either get confirmation or reversion to the mean (i.e. the drug does not work).

I think what I am trying to say is that I suspect the FDA is more focused on weeding out false positives while false negatives are probably (incorrectly) weeded out by business considerations early in the process.

Kind of a weak argument, I agree, but I suspect there is some truth here.

4. Anonymous says:

The failure to fund many of my proposals for new drugs has surely denied people access to therapies that could deliver substantially improved health outcomes.

But it’s not too late. They can still fund me!

3. AE says:

Rova-T is for small-cell lung cancer, not NSCLC.

1. Derek Lowe says:

Right you are! My fingers are just apparently used to typing the “non” in front of that. Fixed.

2. Anders says:

Or “NNSCLC” as we like to call it.

4. johnnyboy says:

The ADC approach is proving to be a lot less simple than it does on paper, isn’t it. Sure it will work on hematologic cancers; but the idea that ADCs will home and concentrate in solid tumors, without any significant degradation, or distribution/capture in bystanders organs, is proving to be much closer to wishful thinking than to reality.

1. Clangordon says:

Agree with ADCs having to have several different components all having to work optimally together to achieve a meaningful benefit to risk profile for patients. DLL3 seems a great target in SCLC but perhaps other treatment modalities targeted to DLL3 will be successful, like CARTs or BiTes.

5. luysii says:

Here is one example of footdragging by the FDA which did hurt people, although it was 50 years ago. USA neurologists in the last half of the 60s read the many papers from Europe about how wonderful L-DOPA was for Parkinson’s disease. We badly wanted to get L-DOPA into the states. The FDA wouldn’t accept the European studies, and made us repeat them. I was an Army doc and I had pathetic patients using their rank as a wedge so they could be entered into an L-DOPA study at the local university. I hated the FDA for years because of it.

Two great things happened in September of 1970
l. I got out of the service
2. L-DOPA was released in the USA

The chief tasked me with running the newly formed L-DOPA clinic. The effects were truly miraculous as our previous therapy was so lousy. I saw one or two patients actually leave their wheelchairs and begin walking again.

1. MrXYZ says:

I am sure there are direct comparisons of EMA approvals and FDA approvals. Any idea of the numbers of drugs that have been approved by EMA but not by FDA and vice-versa (I am only talking about drugs that people tried to get approved in the US and Europe)? You might want to correct that number for any drugs that were approved and then taken off the market for safety reasons (such as thalidomide and rimonabant).

2. luysii, says:

L-DOPA literally was, as Derek says, a “massive, epochal throw-away-your-crutches effect size”

3. JB says:

And here’s an example of foot dragging by the FDA that ended up saving taxpayers billions of dollars and thousands of patients from unwarranted risk: solanezumab. Regulating drugs isn’t easy.

1. Anonymous says:

How can a compound with picomolar affinity for its target NOT be a good drug?

1. Ben T says:

I might have to put that on an exam some day!

4. You have to keep in mind that the period of the late 60’s was just after the thalidomide disaster, which led to the birth of thousands of deformed babies across Europe. Thalidomide (under a variety of trade names) was widely accepted and sold across Europe. However, the FDA official assigned to the case here in the US, Dr. Frances Kelsey, refused to approve it because she thought the toxicology information was insufficient. This was in spite of tremendous political pressure applied by the drugs sponsor. Everyone in the US can be happy and proud that it didn’t get approve here. You can read my blog post about this episode at: https://xconomy.com/national/2015/08/12/dr-frances-kelsey-american-hero-government-regulator/

6. sgcox says:

But luysii, there was another story about that time when FDA did not accept European studies…

1. luysii says:

sgcox — You’re absolutely correct about Thalidomide (which amazingly came back to be used for multiple myeloma — I think they’re presently using an analog). The difference between the Thalidomide and L-DOPA stories, is that there were tons of well performed European L-DOPA studies available; most, as I recall, from academic institutions with no financial stake in L-DOPA.

7. JIA says:

For another example of what Derek is talking about, see: pembrolizumab (Keytruda) in multiple myeloma. This anti-PD1 drug was already approved in multiple cancers; the Phase I/II data (single arm, non-randomized, but decent number of patients) looked promising; the Phase 3’s were started with positive expectations. Then they were shut down in 2017 for increasing — in one trial, increasing by 50%! — the rate of death. Link to FDA announcement, including Kaplan-Meier plots, in my handle.

8. David Young MD says:

Small Cell lung cancer is a tough nut to crack. 8 years ago there was Picoplatin and Amrubicin that were set to be FDA approved as second line…. both failed. Rova-T reminds me of the Bayer’s Matrix Metalloproteinase inhibitor. The concept of inhibiting this enzyme and prevent metastases was big stuff back around the year 2000. But in 2002 Bayer’s MMPI inhibitor actually resulted in increased mortality. After that study came out, the whole concept of MMPI inhibition was dumped. Other than the approval of immune check point inhibitors, the treatment for small cell lung cancer has not changed in 25 years.

9. JIA says:

The history of drug development for MMP inhibitors is interesting and instructive. I recommend a recent review (link in my handle).

10. Anonymous says:

“… 2015. That’s when the first clinical results for Rova-T were reported.”

No one seems to have suggested that Stemcentrx might have misrepresented their results to those dangling the dollars before them. I regret to say that I know of several cases where companies presented (or deliberately misrepresented) early stage (pre-pre-clinical) results to potential purchasers or licensees. In several cases, the bigger companies took the bait and made big mistakes — I mean, big investments — or bought the smaller companies outright.

For example, when a potential lead compound was tested against an important target in duplicate, one result showed 0 (zero) inhibition and the other result showed 100% inhibition. Most Pipeline readers as cynical as I am might say, “Something seems to be wrong. Better re-run the assay and look for potential sources of error.” Those same Pipeline cynics know that management deleted the zero result and presented a “highly potent inhibitor.” After acquisition, the program crashed and burned. Did the buyers care about looking at ALL of the original data and suing for fraud? Nah … there are too many more companies waiting to be acquired!

If there were some lawsuits that rose to the level criminal fraud and some biotech CEOs went to prison, there might a few less scams, fewer billions of dollar going into the pockets of the scammers, and more dollars available to fund more worthwhile research programs (such as mine! – See my other post in this thread).

1. tangent says:

Pie in the sky: a whistleblower clawback mechanism for deals, triggered by deliberate misrepresentation that reaches a certain level. If a company engaged in this, a whistleblower employee can get it rolled back and take a 50% cut.

2. Reality Check says:

Fraud? Conspiracy? Companies (and people) present themselves in the best light everyday and just as in online dating, commonly deployed strategies sort through things quite efficiently. Many preclinical CROs run a tidy business profiling compounds to the nines submitted by investor groups and corporate venture funds as part of their evaluations.

To the point of the article, well designed late stage trials show us the real deal. If other models could do it, they would be used.

11. Lou Wainwright says:

Question from non-industry person. Are there examples of drugs which have significantly outperformed in Phase III? I.e. looked barely good enough to advance from P-I, in P-II it did a bit better, just enough to justify moving ahead, but in P-III it did really really well? Because it seems rational that the flip side of small number variance is that good drugs must sometimes under-perform their true capabilities and don’t always get to move on. I.e. in baseball sometimes a journeyman minor leaguer get a shot in the ‘show’ and turns out to be a star.

Or is this assumption simply not true. Are there so few solid candidates that any half-way decent prospect in P-I is going to get a shot at P-II? In other words is there any evidence that effective new drugs are washing out in P-I due to insufficient testing?

1. Andy says:

That seems like something that should be seen, I hope someone can answer!

2. Lambchops says:

I can certainly think of examples where drugs have outperformed expectations in phase III but nothing is immediately springing to mind in terms of a switch from “barely any effect” to “huge success” but that’s just from my limited experience!

Even drugs that have outperformed expectations come with caveats. The second generation ALK inhibitors for NSCLC spring to mind – if you’d asked clinicians to predict survival outcomes in the latest range of trials their guesses would have been pessimistic . . . but a decent portion of this big increase in survival is due to there being several life extending drugs becoming available and therefore being driven by subsequent therapies as well as the investigational therapy.

On a slightly different tack not an example of a borderline phase II and a successful phase III but an example of hypothesis that may have never been noticed without a large study. There are hints from a phase III study in thrombosis (around 10,000 patients) that canakinumab may have reduced incidence of lung cancer (suggesting a possible role for anti-inflammatory therapy targeting the interleukin 1b pathway in lung cancer) – http://theoncologist.alphamedpress.org/content/23/6/637

3. Lambchops says:

Also this paper (heart disease focused) is worth a read – https://www.nejm.org/doi/full/10.1056/NEJMra1510064

It’s looking at trials which have failed their primary endpoint in phase II and highlights the circumstances in which it may be worth pressing ahead after disappointing results, and emphasises that these circumstances are really rather rare (and most often due to decisions in trial design which were, in retrospect, rather poor).

1. Barry says:

but among the rare drugs that failed their primary clinical endpoint (curiously, it was angina in each case) are minoxidil and sildenafil. Each of which went on to make $billions as “rogaine” and “viagra” 12. Come on, Derek. “Worse than useless” is not the same thing as “worse than the standard of care”. When I saw the headline I was expecting to read about something like the CAST trial, where the antiarrythmics they studied hugely increased mortality compared to placebo. And even as regards those, you can still buy flecainide, so presumably it’s not useless for everybody. 13. JB says: That’s why you *never, ever* use the term ‘efficacy’ for any preclinical data or for any data in clinical trials before phase III. Efficacy cannot be proven until you have a much larger trial, in humans. I cringe whenever people use the term ‘effiacy’ when they shrink a tumor in a mouse model, because they’re just jinxing themselves. 1. steve says: That’s just definitional. You can certainly talk about efficacy in a preclinical (or even in vitro) model. You’re defining the word efficacy as only meaning clinical efficacy when all it means is that it’s efficacious under whatever conditions are being used. It’s up to the reader to understand that a petri dish or a mouse is not a human. 14. Sxa says: Does anyone know where to get a pdf of the green flame? 15. Kathryn Loving says: “clinical failure rates of about 90%”… “There is no substitute for large, well-controlled, well-designed human trials. No one likes to hear that, because everyone’s heard it before, and because it’s not exciting and new and holds out no promise for huge accelerated money-saving breakthroughs.” But it does though… clinical failure rates are high, and if we can’t predict when they’ll happen, we should do more of them, and get there faster, if we want to help more patients. More “shots on goal” if you like sports metaphors. VCs know this model well! Note I’m not suggesting to cut corners on the trials or the INDs at all, but to avoid delaying them because someone wants to run another preclinical non-human-relevant assay because they have a non-evidence-based idea that it will improve their odds of clinical success. Of course the big thing required for “more clinical trials” at a small pharma is money from enthusiastic investors. And people who are focused on speed to the clinic. This was one really unique thing about Stemcentrx: when Abbvie bought them they had filed 5 INDs in about as many years, for different oncology targets, which helped fill Abbvie’s pipeline. Of course we have to wait until all the trials are over to know whether any of them work. But I don’ t think we should view the failure of one drug as a “brutal Greek tragedy” but rather look at the overall approach as something to emulate. 1. Derek Lowe says: It may be, though, that if you file five INDs in five years in a completely new area like this that your clinical failure rate may be far higher than 90%. Something to consider. . . 2. MTK says: Not all “shots on goal” are equal, however. To stretch the analogy, in hockey there’s an inverse square relationship between shot distance and goal expectancy. There’s actually a cottage industry in hockey analytics that tries to measure shot quality. So maybe not all that different from pharma evaluating programs in biotech, except for the fact that each shot on goal in drug discovery costs something. Actually a lot. Not only in terms of money but also in terms of opportunity. Even the largest organizations can only support a finite number of clinical programs. A shot on goal in one program means one other that doesn’t get a chance (or shot if you will). In that type of environment then you better do your damnedest to make sure that those shots you take are high quality ones. 16. M.Striker says: Quoting Derek: “That’s because the early trials involve carefully selected patients, but it’s especially because the early trials are very small.” Can this level of selection still prove a valid market? Or is that also out? (My reasoning being that “blanket solutions are few, and far between”. Also based on my experiences with psych meds that “do X for everyone (/most people)”.) 1. loupgarous says: The problem being that studies with numbers of patients that give them real statistical power will necessarily grow fewer as (a) the cost of providing new drugs increases or (b) the indications for use of the drugs are rarer. Sooooo… when you have something like small-cell lung cancer in which it’s reasonable to pay billions (with a “b”) for the rights to develop and sell for promising new drugs, that’s where drug development money gravitates. Investors expect big returns, after all, and a drug that costs a few hundred a dose to make and can be billed for a few hundred thousand a course of treatment sounds nice to the bean counters, and they raise big venture capital and write big checks. It tends to work where the drug’s mechanism of action is clear, and big drug capital is needed to develop it into a therapy. That’s probably the case with radiooctreotates for neuroendocrine tumors, where you have a proven mechanism of action. Octreoscan using indium-111 bound to octreotide has been an accepted imaging test for neuroendocrine tumors for well over 20 years, so the obvious step up is putting a radionuclide with an acceptably short half-life and more energetic radiation per mole on the octreotide molecule to get that up close and personal with the tumors (in some neuroendocrine tumors, tumors large enough to visualize probably are already metastatic, according to some folks at places like Duke, where they’re seen a lot of them). FDA just approved “Dotatate” (lutetium-177, which is both a beta and gamma-emitter, bound to octreotide) for neuroendocrine cancer in spring 2018, and Novartis bought the company that developed it. THAT is how management and investors in Pharma would love for it to work. And work, it does – my tumors had begun to spread and grow in size around 2010 (fortunately, they took their time doing it), so when I entered the clinical trials for Lu-177 octreotate in 2014, I was a logical test subject. And my tumors became mostly mitotically inactive, my cancer regarded “stable” until winter 2017, when we saw distant metastases all over the place. That’s actually a good run for my sort of disease, which flouts every “rule of tens” in neuroendocrine oncology. So, more radiooctreotate, and I find out in a week and a half if it’s worth the$70,000 a dose afrer two doses (half the four-dose course in the protocol for that therapy).

I’m grateful to have had access to the therapy at all, and for the chance to have helped advance the standard of care for a nasty, if slow-growing, form of cancer. We’ve come a long way from, when I was still working in Pharma, I was diagnosed “cured” because the oncologists of Indiana and my employers’ HMO sagely agreed that the pain I was still in after a five-centimeter tumor was cut away from near my heart and spleen wasn’t due to cancer. After all, the MIBG scan (negative in my case) was standard-of-care, and octreotate imaging a needless expense. It took discovery of three large solid mets by oncologists with a little more savoir-faire in my liver eight years later to shoot a hole in that theory. Now we have better tools and more knowledge.

2. Barry says:

the difference between Tarceva and Iressa is less a matter of structure (although Tarceva is just different enough not to tread on the Iressa patent) than in the selection of the clinical population against which they were tested. Genentech carefully genotyped the cancers for Phase III and got a finding of “efficacious”. Iressa failed against the same ostensible indication.

17. Kelvin says:

This happens all the time because project teams design trials to prove that their drug works in their artificial setting, rather than testing *whether* their drug will work in the real setting. Progress seeking with bias, rather than truth seeking with scientific curiosity. If your commercial forecasts make certain assumptions (implicit or explicit), then the goal of development should be to test those assumptions (hypotheses), starting with the most unreliable.

1. loupgarous says:

It might be time for FDA to study how the Securities and Exchange Commission looks at “forward looking statements”, and for the SEC to study the wreckage from things like Elizabeth Holmes’ Theranos for early clues that start-ups are more efficient at sopping up venture capital than producing new treatments and/or tests for illness.

But it’s hard to see how we get away from in vitro testing to prove out a concept before going ahead to later phases of drug development research, if that’s what you’re saying we should do.

Institutional review boards are going to want to see that in vitro proof of concept before authorizing animal model studies, much less “the real setting” with human volunteers, then with subsets of the patient population.