Skip to main content

Clinical Trials

A New Look at Clinical Success Rates

Andrew Lo of MIT and his co-workers have published a really interesting paper on clinical trial probability-of-success numbers. It appears to be the largest such effort yet:

In this article, we construct estimates of the POS and other related risk characteristics of clinical trials using 406 038 entries of industry- and non-industry-sponsored trials, corresponding to 185 994 unique trials over 21 143 compounds from Informa Pharma Intelligence’s Trialtrove and Pharmaprojects databases from January 1, 2000 to October 31, 2015. This is the largest investigation thus far into clinical trial success rates and related parameters. To process this large amount of data, we develop an automated algorithm that traces the path of drug development, infers the phase transitions, and computes the POS statistics in hours.

They have some 400,000 data points to work with, roughly one-third of which are associated with industrial drug development. About 15% of the large set also had no termination date associated with the trials, so median lengths were imputed, and trials were marked as failed if no further action was observed after defined intervals. They count a trial, very reasonably, as the investigation of a particular drug for a single indication. If a trial is terminated early for any reason except early positive data, it’s marked as failed, and if a drug makes it through one phase and does not move on to the next, it’s listed as “terminated in Phase X”. A difference between this paper and others is that they’re trying to get “path by path” numbers, teasing out individual drug projects and counting them up, as opposed to finding (say) the total number of Phase II trials that started in a given period (a “phase by phase” approach, as the paper has it). As they point out, this is really only possible in more recent years when registration of trials has become mandatory (the data set itself, though, covers 2001-2015, and registration became mandatory in 2007).

They come out with higher success rates than the other studies in this area. The standard estimates for overall probability of clinical success is about 10%, but this study has 13.8% of all pathways  actually making it through. The biggest difference is in the Phase II-Phase III transition, and this is thought to be due to better coverage of missing trials.

A closer look at the data, though, tells an even more different story. That overall POS figure is heavily dragged down by low success rates in oncology. Of the 41040 total pathways in the set, 17368 are for oncology (note that the same drug tried against two different types of cancer will show as two different pathways). The POS of everything outside of oncology is 20.9%, which the POS in oncology itself is 3.4%. If you look at lead indications, instead of all indications, the POS goes up overall (which is in line with earlier studies). But the Phase 2 to Phase 3 transition rate actually goes down a bit, interestingly. Oncology is still the lowest of bunch.

The authors tried to see if biomarkers are helping out (since they’re supposed to). Only 7% of the trials used a biomarker at all stages of development – some used them only for patient selection at the start, for example. Almost all the biomarker-using trials (of any kind) are post-2005. Of the trials that use them to stratify patients at the start (which are almost all oncology trials), the POS nearly doubles, which is good to see. But the broader picture is messier:

However, when we expanded the definition of a biomarker trial to include trials with the objective of evaluating or identifying the use of any novel biomarker as an indicator of therapeutic efficacy or toxicity, in addition to the selection of patients, we obtained significantly different results (see Table S3 in Section A6 of the supplementary material available at Biostatistics online). Instead of finding a huge increase in the overall POS, we find no significant difference. It may be that trials that attempt to evaluate the effectiveness of biomarkers are more likely to fail, leading to a lower overall POS compared to trials that only use biomarkers in patient stratification. Comparison of the two tables shows that new biomarkers are being evaluated in all therapeutic areas.

What about orphan diseases? Using the NIH and EU definitions, success rates are lower in every way in these. Over half the trials so classified are in oncology, and their POS is a hair-curling 1.2%. If you get rid of all the oncology pathways, the POS for “orphan everything else” is 13.6%

Interestingly, when the paper considers POS over time, it appears that success rates decreased from 2005 to 2013, and then picked back up a bit. This is a direct effect, naturally, of the increase in FDA approvals in the last few years, since that’s how success is measured. The graph makes things look more dramatic than they really are, since 2015 is a boundary of the study data – all you can say is that 2014 was a bit better than 2013, reversing a years-long trend, and that 2015 was also an improvement. It’s worth noting, as the authors do, that recent jumps in the immuno-oncology field are having an effect (Nivolumab, for example, was approved five times between June 2014 and June 2015 for different indications). And here are the figures on the length of all these trials:

We find that the median clinical trial durations are 1.6, 2.9, and 3.8 years, for trials in Phases 1, 2, and 3, respectively. Our findings for Phase 3 are higher than Martin and others (2017), but lower for Phase 1. The clinical cycle times for Phase 2 trials are similar. By summing up the individual durations across Phases 1 through 3 and across therapeutic area, we find that the median time spent in the clinic ranged from 5.9 to 7.2 years for non-oncology trials, but the median duration for oncology trials was 13.1 years. This suggests higher risks in oncology projects and may explain their lower approval rate.

That oncology figure surprises me – one of the things about cancer trials is that they’re supposed to move along compared to a lot of other therapeutic indications. Perhaps this indicates a lot of “Well, let’s try this other indication, then”, which the field is well known for. There’s another figure that I wanted to highlight as well – the paper finds (as have others) that the POS increases when there are industrial collaborations in the clinic with non-industry partners (academia, foundations, etc.) The authors use this to note the benefits of such collaborations – which is clearly true – but it needs to be noted that there’s a real selection bias involved. It does not mean that we can raise our success rates by always including non-industry partners. The success rates are increased because of the kinds of projects/targets/drugs that tend to be run in this fashion, more likely, and what we could use is more of those.

Overall, this paper is a very solid contribution to the hard data on clinical trials. The differences between it and other studies are going to be the subject of some arguments, and I look forward to watching those play out. But the authors have gone to a lot of trouble to try to produce the best look yet at the process, and these findings definitely can’t be ignored. There will be disagreements on the way they’ve worked out their pathway data and the usefulness of that versus the “phase by phase” approach (which gives lower success rate numbers).

But I think the issue that I’d like to see getting the most play is the one that’s probably the hardest to argue with: the way that these numbers show the sheer oncologiness of drug development in general. We can argue about what the “right” figure should be, but there seems no doubt that half or more of the drug discovery and development work in the US and EU is going towards cancer indications.

17 comments on “A New Look at Clinical Success Rates”

  1. Dave says:

    What is the definition of probability of success used in this study?

    1. Derek Lowe says:

      Success is defined as an approved drug/indication combination by the FDA.

      1. David says:

        Huh – so if a drug goes through phase 1, 2, and 3 trials, but fails to receive FDA approval, does that count as one failure or three?

        1. Todd Knarr says:

          One failure, the way I read it. They’re counting unique paths from compound through the clinic to approval, so if the same compound went back into phase 3 for a different indication and failed to get approval it’d count as 2 total failures: 1 failure for the first path, 1 failure for the second.

      2. Richard Bernstein says:

        Many trials are positive in a way that changes medical practice for the better, but doesn’t lead to a new fda indication.

        1. Derek Lowe says:

          True, but most of those are not the trials of new chemical/biological entities, I think.

  2. Andy II says:

    I wonder if they had set appropriate categories: 1) Clinical trials for new approval and 2) Clinical trials for label expansion(s) or different indication(s) for the already approved drug. Look at the current clinical trials. Many are label expansions (additional indications) for anti-PD-1 with combination.

  3. Mike says:

    What I’d be interested to see is where the failures are distributed along the development pathway. Derek is correct in that oncology is generally felt to be a much safer bet, but the success rate looks pitiful here.

    What I’m wondering is if oncology failures are front-loaded. In other words, they are failing much earlier (and before much money is spent) than other therapeutic areas. As a result, by the time these compounds start to get people’s attention, they’ve already passed through the riskiest part of the pathway. After that, the success rate is much higher so on the surface it looks like oncology is a safer bet.

    Anecdotally, you tend to see a lot fewer oncology “crash and burn” failures (big phase 3s) than in other disease areas.

    1. tt says:

      The reason for fewer big phase 3 clinical failures in oncology is due to the fairly clear clinical readouts you can get in Ph2 oncology studies as well as that these drugs are not planned for chronic dosing (hence lower bars on safety and side effects that doom other therapeutic areas, like for neuroscience, immunology, CV, etc…). It also seems reasonable that oncology failures are front loaded as it’s fairly easy to get proof of mechanism in patients in Ph1 or in good model studies, on the other hand, a lot of failures in oncology are due to lack of a durable response…i.e. early on you see proof that the tumor is being effect, but over time the outcome versus standard of care is worse or no better…hence it’s on to the next tumor type/tissue/patient pop with the same MOA. Trial and error and error and error…

      1. David says:

        Am I right in thinking it’s also true that approvals for cancer drugs get a bit of a curve compared to other indications, vis-à-vis efficacy and side effects? It’s my impression that some drugs’ efficacy in cancer is as small as 25% chance of 6 months increased average survival or 10% better reduction /inhibition in tumor size/growth, and that serious side effects aren’t as big a concern.
        A skin rash cream, anti-itching agent, or mild pain reliever, however, that results in 10% improvement over control/standard treatments, particularly with off-putting side effects. would seem to be a comparative non-starter.

        I suppose what I’m saying that my impression is that some cancer drugs only need demonstrated, if negligible/minimal efficacy while tolerating significant side effects, whereas other classes of drugs (eg for less terminal indications) have a much higher bar to pass, needing to much more completely alleviate/cure than cancer drugs.

  4. MikeC says:

    “That oncology figure surprises me – one of the things about cancer trials is that they’re supposed to move along compared to a lot of other therapeutic indications.”

    Could the delays be due to difficulties in enrolling enough patients? I’ve read (in comments on this blog, no less), that this is a particular problem in oncology trials.

  5. Mark Plummer says:

    I guess the number 21 143 compounds jumps out at me… how can anyone count compounds in this manner? Truly deceptive

  6. I’ve only just accessed the full paper and interested to see if and how the authors account for abandonment for economic reasons.

  7. PorkPieHat says:

    Is anyone else surprised by the lower POS in Orphan Diseases? I had believed that the typically genetic basis of most rare diseases would provide more a more specific therapeutics strategy that increases the POS in orphans. From their paper:
    “Our data reveal that most orphan drug trials are in oncology. Our overall POS of 6.2% is much lower than the 25.3% reported in Thomas and others (2016). This discrepancy can be attributed to their identification of only non-oncology indications as ‘rare diseases’ and their use of the phase-by-phase method of computing the POS. Our estimated orphan drug POS increases to 13.6% after excluding all oncology indications from the calculations, which is more in line with the findings of Thomas and others (2016).”
    Still, the disparity between 13.6% and 25.3% is large. There must be something else going on besides the poor POS in oncology dragging down their number relative to Thomas and others (2016), right?

  8. Joy says:

    Entirely naive assumption from an outsider: don’t oncology trials often enroll patients for whom the trial substance is a “last resort,” i.e. patients who are in later stages and possibly with more refractive disease? Seems that might be one of the factors lowering success rates in oncology.
    My only knowledge of such trials is as a family member of a patient, but that’s how it looked from where I sat.

  9. tangent says:

    Nivolumab, for example, was approved five times between June 2014 and June 2015 for different indications

    Is that counted as five “paths” each ending in a success? If that’s the way, then if it it were disapproved for those indications, that would presumably be five paths each ending in failure. Or they might got four failures and then a success.

    “Salami-slicing” into indications seems like it could explain the large amount of oncology in here, and its low success rate. When a drug gets shotgunned across some indications:
    bafflegabumab – left pinkie toe cancer – rejected
    bafflegabumab – left ring toe cancer – rejected
    bafflegabumab – left middle toe cancer – rejected
    bafflegabumab – left pointer toe cancer – rejected
    bafflegabumab – left big toe cancer – approved!
    There’s a proliferation of oncology in the data. And this is a successful /drug/ — if 100% of oncology drugs were like this, you’d be elated — but the path success rate in the data is low.

    (Path success rate matters, because each attempt at an indication presumably takes new clinical data that costs money. But you get to reuse the time and money that went into the underlying compound.)

    So “seems no doubt that half or more of the drug discovery and development work in the US and EU is going towards cancer indications” — perhaps half or more of the development but a smaller fraction of the discovery?

  10. Lewis Perdue says:

    What % of failed studies can be attributed to the fact that the animal model is too genetically dissimilar to humans?

Comments are closed.