Andrew Lo of MIT and his co-workers have published a really interesting paper on clinical trial probability-of-success numbers. It appears to be the largest such effort yet:
In this article, we construct estimates of the POS and other related risk characteristics of clinical trials using 406 038 entries of industry- and non-industry-sponsored trials, corresponding to 185 994 unique trials over 21 143 compounds from Informa Pharma Intelligence’s Trialtrove and Pharmaprojects databases from January 1, 2000 to October 31, 2015. This is the largest investigation thus far into clinical trial success rates and related parameters. To process this large amount of data, we develop an automated algorithm that traces the path of drug development, infers the phase transitions, and computes the POS statistics in hours.
They have some 400,000 data points to work with, roughly one-third of which are associated with industrial drug development. About 15% of the large set also had no termination date associated with the trials, so median lengths were imputed, and trials were marked as failed if no further action was observed after defined intervals. They count a trial, very reasonably, as the investigation of a particular drug for a single indication. If a trial is terminated early for any reason except early positive data, it’s marked as failed, and if a drug makes it through one phase and does not move on to the next, it’s listed as “terminated in Phase X”. A difference between this paper and others is that they’re trying to get “path by path” numbers, teasing out individual drug projects and counting them up, as opposed to finding (say) the total number of Phase II trials that started in a given period (a “phase by phase” approach, as the paper has it). As they point out, this is really only possible in more recent years when registration of trials has become mandatory (the data set itself, though, covers 2001-2015, and clinicaltrials.gov registration became mandatory in 2007).
They come out with higher success rates than the other studies in this area. The standard estimates for overall probability of clinical success is about 10%, but this study has 13.8% of all pathways actually making it through. The biggest difference is in the Phase II-Phase III transition, and this is thought to be due to better coverage of missing trials.
A closer look at the data, though, tells an even more different story. That overall POS figure is heavily dragged down by low success rates in oncology. Of the 41040 total pathways in the set, 17368 are for oncology (note that the same drug tried against two different types of cancer will show as two different pathways). The POS of everything outside of oncology is 20.9%, which the POS in oncology itself is 3.4%. If you look at lead indications, instead of all indications, the POS goes up overall (which is in line with earlier studies). But the Phase 2 to Phase 3 transition rate actually goes down a bit, interestingly. Oncology is still the lowest of bunch.
The authors tried to see if biomarkers are helping out (since they’re supposed to). Only 7% of the trials used a biomarker at all stages of development – some used them only for patient selection at the start, for example. Almost all the biomarker-using trials (of any kind) are post-2005. Of the trials that use them to stratify patients at the start (which are almost all oncology trials), the POS nearly doubles, which is good to see. But the broader picture is messier:
However, when we expanded the definition of a biomarker trial to include trials with the objective of evaluating or identifying the use of any novel biomarker as an indicator of therapeutic efficacy or toxicity, in addition to the selection of patients, we obtained significantly different results (see Table S3 in Section A6 of the supplementary material available at Biostatistics online). Instead of finding a huge increase in the overall POS, we find no significant difference. It may be that trials that attempt to evaluate the effectiveness of biomarkers are more likely to fail, leading to a lower overall POS compared to trials that only use biomarkers in patient stratification. Comparison of the two tables shows that new biomarkers are being evaluated in all therapeutic areas.
What about orphan diseases? Using the NIH and EU definitions, success rates are lower in every way in these. Over half the trials so classified are in oncology, and their POS is a hair-curling 1.2%. If you get rid of all the oncology pathways, the POS for “orphan everything else” is 13.6%
Interestingly, when the paper considers POS over time, it appears that success rates decreased from 2005 to 2013, and then picked back up a bit. This is a direct effect, naturally, of the increase in FDA approvals in the last few years, since that’s how success is measured. The graph makes things look more dramatic than they really are, since 2015 is a boundary of the study data – all you can say is that 2014 was a bit better than 2013, reversing a years-long trend, and that 2015 was also an improvement. It’s worth noting, as the authors do, that recent jumps in the immuno-oncology field are having an effect (Nivolumab, for example, was approved five times between June 2014 and June 2015 for different indications). And here are the figures on the length of all these trials:
We find that the median clinical trial durations are 1.6, 2.9, and 3.8 years, for trials in Phases 1, 2, and 3, respectively. Our findings for Phase 3 are higher than Martin and others (2017), but lower for Phase 1. The clinical cycle times for Phase 2 trials are similar. By summing up the individual durations across Phases 1 through 3 and across therapeutic area, we find that the median time spent in the clinic ranged from 5.9 to 7.2 years for non-oncology trials, but the median duration for oncology trials was 13.1 years. This suggests higher risks in oncology projects and may explain their lower approval rate.
That oncology figure surprises me – one of the things about cancer trials is that they’re supposed to move along compared to a lot of other therapeutic indications. Perhaps this indicates a lot of “Well, let’s try this other indication, then”, which the field is well known for. There’s another figure that I wanted to highlight as well – the paper finds (as have others) that the POS increases when there are industrial collaborations in the clinic with non-industry partners (academia, foundations, etc.) The authors use this to note the benefits of such collaborations – which is clearly true – but it needs to be noted that there’s a real selection bias involved. It does not mean that we can raise our success rates by always including non-industry partners. The success rates are increased because of the kinds of projects/targets/drugs that tend to be run in this fashion, more likely, and what we could use is more of those.
Overall, this paper is a very solid contribution to the hard data on clinical trials. The differences between it and other studies are going to be the subject of some arguments, and I look forward to watching those play out. But the authors have gone to a lot of trouble to try to produce the best look yet at the process, and these findings definitely can’t be ignored. There will be disagreements on the way they’ve worked out their pathway data and the usefulness of that versus the “phase by phase” approach (which gives lower success rate numbers).
But I think the issue that I’d like to see getting the most play is the one that’s probably the hardest to argue with: the way that these numbers show the sheer oncologiness of drug development in general. We can argue about what the “right” figure should be, but there seems no doubt that half or more of the drug discovery and development work in the US and EU is going towards cancer indications.