Skip to main content

Cardiovascular Disease

Clinical Trials: Getting Much Worse, or Much Better?

Here’s a very interesting overview of the clinical trials funded (55 total) by the National Heart, Lung, and Blood Institute (NHLBI) over the years 1970-2012. Breaking them into pre-2000 and post-2000 shows something dramatic: in the 30 older trials, 57% of them reported a positive, beneficial outcome. Of the 25 newer ones, only 8% came back positive. Comparisons to placebo, trial design, and so on were quite similar between the two sets – so what the heck? Has cardiovascular disease gotten a lot harder (the “low-hanging fruit” effect)? Have clinical investigators, or the people providing them the money, become a lot less competent (let’s hope not)?

Neither one, most likely (or at least not enough to skew the numbers that wildly). No, the authors picked the year 2000 because that’s the year that NHLBI trials starting having to be registered on, and thus to declare up front what the goals of the trial were.

Prospective declaration of the primary outcome variable is important because it eliminates the possibility of selecting for reporting an outcome among many different measures included in the study. In order to investigate this issue, we looked at the statistical significance of other variables not declared as the primary outcomes for preregistered studies. Among the 25 preregistered trials published in 2000 or later, 12 reported significant, positive effects for cardiovascular-related variables other than the primary outcome. Importantly, almost half of the trials might have been able to report a positive result if they had not declared a primary outcome in advance. Had the prospective declaration of a primary outcome not have been required, it is possible that the number of positive studies post-2000 would have looked very similar to the pre-2000 period.

That seems to be it. You can look at this from a glass-half-empty perspective (maybe a lot of those earlier “successes” were cherry-picked illusions), or a glass-half-full one (this whole effect is just an artifact, and trials are finding good results just as much as always). I lean a bit more towards the first, myself – not all the way, but enough to make the call without too much difficulty. This is a stark illustration of what it means to call your shots in the clinic. It affects the design of the trial at the beginning, of course, but it also keeps you from kidding yourself at the end.

10 comments on “Clinical Trials: Getting Much Worse, or Much Better?”

  1. John Tucker says:

    I think there is a fundamental problem with the assumption that the “true” underlying success rate in both periods was the same, and the difference in reporting is due to “fudging”.

    If you look at the supplementary material, one not unexpectedly finds many breakthrough studies of blood pressure control, statins, and the like in the pre-2000 period. In contrast, the majority of the post-2000 trials consist of

    1) Dietary supplements such as vitamins for prevention of MI and stroke
    2) “More of the same” studies, in which treatment with multiple blood pressure or lipid lowering drugs is compared to monotherapy – such studies are intrinsically subject to diminishing returns
    3) Five of the 15 post-2000 studies address a single therapeutic modality – estrogen replacement therapy. This single failure accounts for one third of the trials in the post-2000 period, and ought to have been counted only once.

  2. The latter group also includes the niacin and fibrate studies. I’m suspicious the result could be an artifact. I feel we need more data.

  3. Kelvin Stott says:

    This just fits with what we have seen with diminishing success rates across ALL of drug development over the years. No surprise at all.

    But as for the cause, this is actually very simple: Prioritization.

    Prioritizing opportunities by expected return (as we all do) gives you exactly what you would expect: higher success rates and returns at first, followed by diminishing success rates and returns later.

  4. Amanda says:

    might have already been noted elsewhere, but the link color looks really similar to the text color (hard to see clickable text)

  5. Diver dude says:

    Having spent my industry career from the mid 80s to 2010 designing and running clinical trials across a wide spectrum of therapeutic areas, I have no hesitation at all in saying that I think Derek’s conclusion is correct and, if anything, he is far, far too charitable.

    1. john tucker says:

      Who “designs clinical trials across a wide variety of therapeutic areas?” It’s a function usually reserved for those with a high level of subject matter expertise in the therapeutic area in question. Has one of the grandmasters of drug development appeared in our midst under the moniker “surfer dude”?

  6. I ask questions, I don't do chemistry says:

    “Trials were included if direct costs >$500,000/year”

    Can we infer these are not Phase II trials? Isn’t this the whole “fail fast” thing that everyone thinks is a good idea? Shouldn’t the senior management of the teams whose trials failed feel good about this? They didn’t blow vastly more money on a failure in the next phase.

  7. gippgig says:

    “12 reported significant, positive effects for cardiovascular-related variables other than the primary outcome” – why specifically mention cardiovascular-related variables? Does the NHLBI only run clinical trials for cardiovascular-related disorders?

  8. Alen Mcmilan says:

    Love it! This is a great post. I like this topic.This site has lots of advantage.I found many interesting things from this site. It helps me in many ways.Thanks for posting this again. Thanks a million and please keep up the effective work Thank you so much for sharing this kind of info- clinical research associate training

  9. AlenMcmilan says:

    A clinical research associate training involved in all stages of the clinical trial, including identifying an investigational site and setting up, initiating, monitoring and closing down the trial.

Comments are closed.