Skip to main content
Menu

Clinical Trials

Stopping Early

There was a mention yesterday in the comments section about a clinical trial that was stopped early due to efficacy. I’ve never been involved with a project that this has happened to myself – pretty much the opposite, for the most part! – but it does happen, and is generally cause for celebration.

Although not always. I wanted to link to this blog post by Hilda Bastian that has some examples of times when halting early wasn’t a good idea. Why wouldn’t it be? Well, you can get fooled. Even with a reasonably designed trial, you can get fooled, because statistics are what they are. The post has an excellent quote from an article reviewing an AML trial that was almost stopped early, but looked so odd (oddly good) that it was continued a few months longer. Whereupon the benefit evaporated:

Quite extreme chance effects can and do happen more often than many clinicians appreciate. At any one time, there are hundreds, if not thousands, of trials ongoing, often with analyses at several time points and with a number of subgroup analyses. Thus it is inevitable, with all these multiple comparisons being undertaken, that highly significant results (p<0.002) will sometimes occur by chance and that conventionally significant (p<0.05), but spurious, differences will occur frequently. Taken in isolation, these may well appear so striking to investigators that it will be difficult to believe that these are chance findings. No trial is immune from such random effects, no matter how well designed and conducted.

Absolutely right. Weird readings of this sort are not something that can be designed out; they’re an inevitable feature of running a lot of trials. The equivalent on a smaller scale is rooting through the subgroups of a single large trial and finding that by gosh, this group right over here responded just great: let’s run another trial! We found the people the drug works on! Well, maybe. But if you analyze enough subgroups, the chances of finding a spurious correlation are quite good (particularly since these subgroups all have smaller sample sizes). If you’re not careful, you’ll find yourself realizing that the drug seems to perform best on Libras who own Toyotas, which is not too useful.

Here’s a good example to think about:

To start: a tale of 2 trials of a drug for secondary progressive multiple sclerosis (SPMS) (interferon beta-1b). One started in Europe in 1994; the other got underway in the US and Canada in 1995. The European trial stopped 2 years early after interim results “gave clear evidence of efficacy. Treatment with interferon beta-1b delays sustained neurological deterioration” – the first treatment found to do that for SPMS.

So what then for the North American trial, still in its early stages? The knowledge base providing the ethical justification for their trial had shifted – and they had hundreds of people on placebos. The trial had a data monitoring committee (DMC). The DMC has the role of protecting participants against harm and making judgments about the data during a trial. (A DMC is also called a data and safety monitoring board (DSMB) or data monitoring and ethics committee (DMEC).)

The DMC looked at their data, and decided to keep going. They stopped early, too, in November 1999 – not because of benefit. Unfortunately, there was no benefit on delaying disability. They stopped early for futility – the belief that the outcome wasn’t going to change if they continued. (If you want to brush up on the basics of stopping trials early, I’ve written a primer over at Statistically Funny.)

Where did that leave people with SPMS? Despite 2 trials, the picture was murky. It took another big trial that didn’t stop early to be sure. According to a systematic review in 2011, the evidence that interferon beta-1b doesn’t work for SPMS is “conclusive” (PDF).  (The drug is not approved for the indication of SPMS by the FDA.)

In this case, it’s thought that the first European trial that looked so promising might have admitted too many patients who had not actually progressed to secondary progressive MS. And that is, of course, another way that you can be led down the wrong path. Patient selection and enrollment is a major, major issue. You can produce answers all over the map if you get it wrong. But sometimes you find that getting it right means that you’re either not going to be able to enroll enough patients (or you’re enrolling them too slowly) to meet any kind of reasonable development timeline. An opposite mistake can be made, too: you can greenhouse your patient selection so carefully as to give you a fine trial readout that has little to do with what will happen in the broader patient population your drug will see after approval.

Of course, you can stop early for futility, too, and the same considerations apply – although I will say that since the overall clinical failure rate is 90%, that the odds are better that rosy results will deteriorate on closer inspection rather than a bad result will improve. But it’s not impossible. Run enough trials, and handle enough data, and nothing is impossible. Bastian’s post has other examples, and I highly recommend it. Clinical trial design and interpretation is the most crucial part of this whole business, and it deserves plenty of thought and plenty of respect.

28 comments on “Stopping Early”

  1. anon says:

    May be they saw a quick efficacy during trial and if the study continued they will start to see AE. Hence they stopped! Makes sense?

    1. matt says:

      No, it doesn’t make sense, because if the finding is “effective” then patients will continue taking the intervention, and the AE will happen. And if you are still wondering about AEs in any way, you probably aren’t ready to stop the trial. The only way the possibility of AEs due to your intervention goes away is if you decide to shut down the program.

      I realize this may be sarcasm, but even if you were thinking somebody could “put one over” on the FDA, and even if that were accomplished, the trial lawyers would put you on a spit and barbecue you slowly over the heat of public outrage while all the fat of your ill-gotten gains dripped out into the lawyers’ pans.

  2. Joe psycho says:

    Hi, Derek i am struggling with extremely persistent asthma and do not tolerate β2-agonists and antihistamines that cross the BBB well (they both make me psychotic) prednisone is the best treatment so far I tolerate it unusually well so I think side effects are a small price to pay (munchies and hot flashes vs full blown psychosis)40mg for 3 days and I am like “what asthma” on the third day and feel like superman with the munchies and hot flashes but I know it’s not healthy in the long term but it restores my quality of life and sanity but doctors don’t want to prescribe it (partially due to biologics that they won’t let you self-inject so they can charge you for an office visit with each dose and practically rob you) I personally object to this obvious fraud.
    The big question is what small molecule drugs are in the pipeline for asthma and their MOAs. Would you be willing to help me by posting a list of promising candidates. thank you (the email address is a fake)

    1. Chrispy says:

      Montelukast is not on your list. No one will give medical advice here, but you might want to ask your doctor about this one.

      1. Joe psycho says:

        Was on montelukast it produced a small improvement but it made me very irritable

    2. David Young says:

      … won’t let you self-inject? You mean your insurance will not pay for those medication if you self inject. Don’t blame the physician.

      1. Joe psycho says:

        Doctor wanted money from office visits total scam!

        1. AR says:

          Doctor wants money for his time, scam. Got it

          1. Joe psycho says:

            I am fully capable of self-injection (it’s subQ like insulin) and is just a waste of the doctor’s time and this doctor’s arrogance caused harm

    3. loupgarous says:

      Not a doctor nor do I play one on TV. That said, has your doctor ordered a serum catecholamines/metanephrines assay? Ask your doc why those are done.

  3. Jakob says:

    This reminds me of a study in NEJM published last month:
    “Vitamin D Supplementation and Prevention of Type 2 Diabetes”

    This study actually showes an effect in the early part of the study, where vitamin D supplementation seems to prevent risk of developing T2D, but then in the end, there is no difference.
    Can people develop “resistance” to excessive amounts of vitamin D, so that the effect it lost again after a while?, so perhaps vitamin D can only delay, but not prevent T2D? Maybe vitamin D should be given in controlled cycles like chemotherapy to prevent resistance?
    Anyway, it is nice to see that high impact journals publish such “negative results” also.

  4. Philip says:

    Related: http://www.cardiobrief.org/2016/03/08/controversial-ill-fated-obesity-drug-trial-published-in-jama/

    The interim results were released by the company because they looked good and would raise the stock price. This early release of data poisoned the full study. The interim results did not hold up for the full study.

    An aside, the study was worthless by design. The two arms of the study did not have the same ITT population.

  5. Petros says:

    #Joe Psycho

    Most recent attempts to develop new small molecule approaches to treat asthma have floundered. Novartis’ fevipripant, targeting DP2 (CRTH2) receptors is in Phase 3 for severe asthma and would now appear to be targeted at the same (or similar) segment of the asthma market as the injectables. Most other agents in this class have now terminated development

  6. Marcus Theory says:

    But think of the market share of Libras who own Toyotas!

  7. rtah100 says:

    @JoePsycho

    A colleague with asthma quit steroids for a humanised antibody treatment, only to become one of its FDA black box statistics on the second injection because he turns out to be allergic to Chinese Hamster ovary tissue-derived moieties. His symptoms were masked by the tapering steroids and resulted in an emergency room visit and admittance. The ER doctor thought the whole episode fascinating and offered to administer a third dose “close to resuscitation facilities”. My friend declined. But insisting you get these injections under medical supervision (and my friend’s wife is a cardiologist!) looks less like a scam and more like risk management when you consider the past fatal adverse events and class action litigation.

  8. Joe psycho says:

    Ok then tell me the most effective way to beg a doctor for steroids because all other treatments are checked off the list then

    I tolerate steroids extremely well (only hot flashes, munchies and weight gain) and long term risks can be mitigated very well compared to some other drugs

    And the fact that I cannot get steroids so i can breathe and not be batshit crazy from albuterol and people can smoke freely for no damn reason makes me feel like feeding every smoker to a pit of boa constrictors. if you have any prednisone you want to give me so I can do something other than sitting in front of a screen and getting angry leave a comment to tell me I NEED PREDNISONE my doctors think they are gods I hate most of them.

    1. Some idiot says:

      “And the fact that I cannot get steroids so i can breathe and not be batshit crazy from albuterol and people can smoke freely for no damn reason makes me feel like feeding every smoker to a pit of boa constrictors.”

      (-:

      Magical description…! I think you may need to be an asthmatic to really, seriously understand it (which I am, and do)!

      I really feel for you Joe… I was in a similar situation (and on prednisone) about 35 years ago, but new medication (that I can see doesn’t work for you, very unfortunately) changed my life. Which is why I decided to use my skills in the pharmaceutical industry in order to try to help others in a similar way to what I was helped. Unfortunately, I work in a different area, so I can’t help you, but I really sincerely hope all the very best for you…!

      It sounds like you know what this feels like, so I will mention that at one stage I was in a trial comparing oral and inhaled salbutamol. I am _so_ damned happy that I didn’t have to take oral salbutamol after that!!!

  9. Simon says:

    Joe, get prednisone or 4me pred from online pharmacy w/o prescription.

    I do for occasional use for spider bites etc when I’m in the jungle.

    1. Joe psycho says:

      Can you please tell me what online pharmacy you use, that would be a great help

  10. sufferer says:

    just learn to suffer and try to control environmental factors, move to Arizona. long term prednisone or any steroid use will screw you up.
    There is little can be done, I have really bad eczema, my hands can end up covered in bleeding blisters if I’m not careful, it’s either steroids or suffer, no magic cure.
    Some people find relief by going to vegan diets, maybe they limit protein and inhibit their immune system that way.
    There are a lot of people suffering from these autoimmune diseases

    1. Joe psycho says:

      I would rather die than go completely off steroids I have other diseases that prednisone is also an effective treatment for (autoimmune inflammatory arthritis, chronic urticaria that responds poorly to antihistamines, eczema, severe seasonal allergies, lupus like symptoms but no ANAs ) and the neighbors BURN GARBAGE. I personally hope they die a slow and painful death by lung cancer. I would rather castrate myself with a dull butter knife than suffer through this. When people ask me how are you doing i just want to tell them I have cancer because nobody understands. I honestly would rather have cancer because they understand and doctors provide better care. Anyway my life is hell on earth and I cannot and will not just suffer forever when there is a pill that returns me to a functioning human being, reminds me of the greek myth of tantalus and the fruit and for me that fruit is prednisone.

      1. loupgarous says:

        Your experience parallels mine, before my hypersensitivity to steroids and other strange drug reactions were found to be due to a 5cm secreting paraganglioma located between my heart and spleen. Unfortunately, the physicians on my case were entirely wrong about “well-encapsulated” meaning “benign”. It turned out to have metastasized to my liver and several other places (periaortic lymph node, jaw socket, spine, pelvis among others).

        But what really struck me were the neurological and endocrine responses you have to medications, which resemble signs of secreting neuroendocrine tumors such as pheochromocytomas. It’s worth it, to you, your doctor, and even your insurer, for you to be worked up for neuroendocrine cancer. A short course of PRRT might be cheaper and less painful in the long term than the surgeries and other treatments I underwent before PRRT was developed (I was a clinical trial patient for PRRT).

        1. Joe psycho says:

          My biggest problem is extremely persistent and hard to control asthma.

          1. anon says:

            Joe, I can sympathize with your problem. I have developed also asthma that I believe as a result of mold from a leaky roof. What I find quite surprising is that even after the roof has been reshingled and the moldy dry wall replaced, the asthma still persists. I would have thought that it might eventually recede. My inhales feel quite normal, though it is on the exhales where I can hear and feel my airway shut down. During the day there is no problem; it is more at night and the next morning when I am unsure what apnea happened while I slept.

            Might you have any suggestions for a spirometry that I could buy online that would give me useful datapoints? I hope that you can find an effective treatment for your asthma.

  11. Philip says:

    https://www.washingtonpost.com/world/africa/ebola-no-longer-incurable-scientists-say-after-discovery-of-two-highly-effective-drugs/2019/08/12/d2bcf668-bd40-11e9-a8b0-7ed8a0d5dc5d_story.html

    I hope this early stoppage works out. Seeing a headline of Ebola No Longer Incurable, sounds great. Even if the drugs work well, there are still many roadblocks to getting the sick the needed treatment.

  12. David in Kent says:

    What’s wrong with inhaled steroids like beclomethazone or budesonide? they work topically and without much systemic effect, unlike prednizone.

    1. Joe psycho says:

      I Am on 480μg ciclesonide bid more than the fda maximum dose and spiriva (tiotropium bromide) 5μg (off label dose) and have up to 6 asthma attacks in a day

  13. Joe psycho says:

    Still struggling with asthma and inhaler side effects i can use any help that you may have

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.