Skip to main content


A Painful, Unacceptable Lack of Data

Okay, this paper is not going to make a lot of people happy. The authors are reviewing oncology approvals by the European Medicines Agency (EMA) from 2009 to 2013 – overall, there were 48 drugs approved for 68 indications, which sounds like good news. 24 of the 68 were approved with survival benefit shown in the clinical trials – the rest were approved on progression-free survival or the like. So how has that been working out, post-approval? Data are more scarce than you’d expect, and what’s available isn’t that encouraging:

This systematic evaluation of oncology drug approvals by the EMA in 2009-13 shows that most of the drugs (39/68, 57%) entered the market without evidence of improved survival or quality of life. At a minimum 3.3 years after market entry, there was still no conclusive evidence that 33 of these 39 cancer drugs either extended or improved life. Our findings suggest it is extremely rare for new studies or follow-up analyses of pivotal trials in the postmarketing period to report results confirming that new cancer drugs have a positive impact on the two most important outcomes for patients—survival and quality of life. When survival gains over existing treatment options or placebo were shown, they were often marginal and judged to be clinically meaningful in less than half (11/23, 48%) of all cases.

So roughly half the cancer drugs approved during this period are without this information – it’s not necessarily that these approvals have not led to any survival benefit – it’s that we don’t know if they have or not, because nothing has been reported. It has to be noted (as the authors mention) that only 18 of the indications were supported by trials that had survival as a primary endpoint, and that crossover trial designs often obscure survival in general.

A major factor in all this, of course, is what the regulatory agencies want to see and will accept. Despite a lot of talk about overall survival, the EMA seems to accept a lot of surrogate endpoints as enough for drug approval. And they’ve also been quite generous with approvals based on (basically) any statistically significant difference in survival between treatments, regardless of whether this will be something that looks to hold up in the real world.

Though surrogate endpoints are argued to have the advantage of allowing quicker drug development and patient access, it is questionable whether studies based on surrogate measures of efficacy provide optimal, or even meaningful, information for patients and clinicians. Moreover, our analysis raises the possibility that regulatory and current research practices have created a situation in which critical information about the outcomes that matter most to patients might never be generated once oncology drugs are approved for widespread use.

That’s not a good situation. I’m sure that there will be room to argue about some of the figures in this paper – and there’s plenty of room to argue about whether the situation here in the US, with the FDA, is any better. (There are plenty of arguments waiting when such data become available, too, believe me). But in general, I have to say that I agree with the conclusions that regulatory agencies should be asking for more survival data – if not at approval, then in required postmarketing surveillance. For those cheering this on, though, be aware that nothing comes for free: if you want more solid data on real-world survival, then you’re going to be paying for that, in both money and time. Tightening up the survival requirements will inevitably slow things down, and if you’re in the “Regulations are strangling drug innovation” camp, you’re not going to care for the idea much to start with.

But you pays your money and you takes your choice: faster, cheaper, and less proven – or slower, more expensive, and more proven. The only way I can see to split the difference is as mentioned above: if you’re going to approve on surrogate endpoints, then make it mandatory to collect data on what happens afterwards, and set an appropriate date to review it, with approval to be rescinded if things don’t work out. Sounds good – but remember what happened when the FDA did that with Avastin’s breast cancer indication? Nothing is simple, not with so many lives and so much money on the line.

Update: the EMA has a response to this here – I’ll add more details later, but I wanted to get the link up, because it’s very useful reading.

33 comments on “A Painful, Unacceptable Lack of Data”

  1. PharmaHeretic says:

    So what? Is selling “drugs” with no worthwhile therapeutic benefit wrong if the corporation doing that can make billions of dollars? Look at the bright side.. these drugs were developed by the “smartest” people educated in “ivy league universities” and employed by “world class corporations”. The “free market” has spoken.

    All those peasants who derive no therapeutic benefit from such drugs should he happy that they, or whoever pays for them, are at least benefiting from something developed by the “scientific method” (as defined by modern corporations and universities) as opposed to placebos or older and very unfashionable drugs that give the same therapeutic benefit buy cost much less.

  2. PharmaHeretic says:

    Correction to previous reply is as follows: same therapeutic benefit ‘but’ cost much less.

  3. DCRogers says:

    So, anyone know how many oncology drugs have been approved by the FDA with something other than survival benefit shown in clinical trials?

    The sad thing is, allowing oncology drugs without such data to enter the market punishes companies that *do* provide it – they pay with a larger chance of trial failure and a longer time to market. Any sensible company would switch to the sloppiest surrogate marker available.

    The free-est market in the world, with bad price signals baked into the rules, is destined to go off the rails.

    1. John Wayne says:

      “The sad thing is, allowing oncology drugs without such data to enter the market punishes companies that *do* provide it – they pay with a larger chance of trial failure and a longer time to market. Any sensible company would switch to the sloppiest surrogate marker available.”


  4. Anon says:

    Shouldn’t this be filed under “snake-oil”?

    1. MoMo says:

      All of chemotherapy with the exception of infectious diseases should be labeled under the heading of “snake oil”.

      1. Anonymous says:

        MoMo said:
        All of chemotherapy with the exception of infectious diseases should be labeled under the heading of “snake oil”.
        This is incorrect, chemotherapy for diseases like Hodgkin’s lymphoma results in 10 year survival rates of 80%. “Cure” in 4 out of 5 patients is the opposite of snake oil.
        Chemotherapy for metastatic solid tumors has a less favourable benefit-harm ratio, but don’t forget its efficacy in blood cancers particularly.

  5. Flem says:

    “Faster, Cheaper, and Less Proven”… It may be faster and less proven but it is not cheaper for the one who pays. It is cheaper for the one who sells.
    No one should pay full price for a product that hasn’t demonstrated full value. Maybe sales should be put in escrow if approved before survival data is available – to be returned if survival not demonstrated.

    1. tangent says:

      I do like the escrow. Sensible and vicious.

      1. a. nonymaus says:

        “No cure, no pay” works in marine salvage, maybe it can work here:

        1. Design Monkey says:

          Under “no cure, no pay” method that baking soda charlatan simonchini would be quite rich. No cure happened – charlatan is out of 5 cents worth of baking soda, no biggie. Accidental natural remission = looks like sort of “cure” – charlatan rakes in wheelbarrow of dough.

  6. Anonymous says:

    People develop resistance to some drugs so categories need to change during treatment. One example is Gleevec (approved in 2001). Works great for a while. Then some cancers develop resistance. If so, stop using Gleevec and switch to something else.

    Another big post-approval study involved EPO (erythropoietin) from Amgen (approved in 1989). I wish I had access to the literature, so this is from memory. After ~20 years of use and more than $50 billion in sales (or profits? I don’t remember which.), studies concluded that EPO was not only NOT beneficial when used as an adjunct in cancer chemotherapy but that it might even be causing more harm than good. (It is still of value for the treatment of some anemias and sometimes winning Le Tour de France, but not with chemo.)

    (Most of the above is from memory and Wikipedia. Please check other info independently.)

  7. DH says:

    I suspect that part of the lack-of-data problem is that it’s really hard to get *clean* data. I just had a relative die after fighting cancer for about 3 years. During this time, he was on at least 4 different chemotherapy programs. One program would shrink the tumors for awhile and then stop working, so they switched to something else. Another would seem to work, but he just couldn’t tolerate it (nausea, weight loss, etc.), so they stopped for awhile to let him regain strength and then switched. And so on.

    I imagine that this pattern is more common that not. How in the world can you get clean statistics on any of these treatments when this is how they get used in the real world?

  8. steve says:

    The history of oncology is that, after drug approval, physicians figure out the best combinations over time. Drugs that may not be particularly effective as single agents can synergize with other drugs and yield very effective combinations. So the more drugs on the market the better and let the docs sort them out. Insisting that single agents show significant effects in pre- or post-market surveys to be allowed is just shooting ourselves in the foot.

    1. Some idiot says:

      Very valid point. AIDS drugs form a similar case.

      1. Nyquil says:

        And some CNS drugs too. Look at the ‘once a month’ antipsychotics.

  9. Anonymous says:

    Follow-up on Amgen – Epogen comments. I still can’t figure out profits vs sales. In some years, Epo sales were over $10B, so I think $50B over 22 years might be profits, not sales. See: “Anemia drug made billions, but at what cost?”, Washington Post, July 7, 2012. “[R]esearchers issued an 84-page study declaring that among most kidney patients, the original and largest market for the drugs, there was no solid evidence that they made people feel better, improved their survival or had any “clinical benefit” besides elevating a statistic for red blood cell count.” … “The trouble, as a growing body of research has shown, is that for about two decades, the benefits of the drug — including “life satisfaction and happiness” according to the FDA-approved label — were wildly overstated, and potentially lethal side effects, such as cancer and strokes, were overlooked.” Et sim.

    In reply to another comment, some physicians may just wing it, but there are also continuing studies of some drugs that have been in use for years and are now generic. Periodically, recommended clinical protocols get updated. You need a lot of patients at a large cancer center to determine the difference between “single bolus days 1 and 10” versus “smaller daily doses for 10 days” but they do try to figure out if there is a benefit. After another five years and thousands of patients and followups, they may change the recommendation again.

    But there is still a need for better data from the initial trials themselves, not just the appearance of possible benefit, other than more profits, a better IPO or M&A, or CEO compensation package.

  10. wei says:

    Reading this piece is just like reading arguments that Pharma does not do R&D or Pharma just picks out low hanging fruits from research funded by public. Maybe we should not blame people outside pharma to think this way.

  11. James says:

    Off topic but worth noting– watching the MLB and NFL games today, im reminded how laughably absent basic notions of teamwork and social cohesion are from science compared to these comparatively simple endeavours. I could be incorrect of course and im speaking generally, but i assume the reason for this is that scientists dont want to necessarily accomplish anything of objective import, but just want to beat the next guy that might steal their grant or brownie points from the boss.

    1. Mister B. says:

      Our society is based on this. How to be more competitive than the one next to you ? Animals also do the same…
      Works for companies, scientists, athletes …

      It is a wholesome behaviour if it drives the whole group of entities (companies or men) to higher achievements.

      1. James says:

        You missed the point big man, i was not arguing that competitive behavior is bad. If you want to argue that teamwork and social cohesion ( a dismal lacking of which is present in science currently) then go ahead. Everyone who has acheived something will laugh at you.

    2. Passerby says:

      Most of team sports is training in irrational jingoism, this whole idea that “my team” is somehow superior to everyone else and must crush everyone else. Thanks, but we certainly don’t need more of that attitude in science.

      1. James says:

        All rhetoric aside: Let me be clear–i witnessed brutality in grad school that would make you wake in the night. Deaths, sexual abuse, and naked harassment. In a top 5 us wirld report school 8n chemistry. My guess is that if this was in the NFL/NHL/MLB, there would a massive scandal. So yeah, my point is clear. Scientists are not good people, in general.

  12. Emjeff says:

    I find the EMA’s response pretty convincing. I think this is a space where the true benefit of new oncology drugs on survival will probably come out in large consortium trials. These trials take time and money, though, so do we not approve until these survival data are in ( which could take many years) or are we up-front with patients about what to expect from these new drugs and allow them to make an informed choice? I think the latter approach is the humane one.

    1. Chief Ignorance Officer says:

      So, you are saying we approve drugs without proof of efficacy and we see if we can get it later on in the marketplace?
      If we accept this, why run efficacy trials at all? Just approve the damn drug based on safety, as is done in some places?

      On the other hand, is it ethical to charge 6-figures prices for something that just *may* work?

      1. Some idiot says:

        Nope, that’s not what he is saying. What he is saying is that if you can get some indication that the stuff is beneficial, then if the data is good enough then it can be put on the market while survival rates get worked out. Then the market will decide whether or not it is useful (and whether or not it is worth some astronomical sum).

        Which is not the same as “not doing efficacy”. I think Steve’s comment above is also very relevant here.

        1. Chief Ignorance Officer says:

          For what it is worth, I still don’t buy it.
          One thing is to show robust efficacy in one area, and then do combination trials to expand it to additional indications. It will work sometimes, and sometimes it won’t. I believe one such examples is keytruda.

          Another thing is to have a drug approved based on no benefits shown, and hope that later on areas of acceptable efficacy will be found.

          1. Some idiot says:

            Fair point. I think the main point is coming up with a good, useable definition of “Robust efficacy” here, which balances appropriately between very clear-cut exhaustive data/long-term survival rates on the one hand, and a reasonable time to market on the other (and I accept that the term “reasonable” here is just as slippery as anything…).

            Put another way, I believe that it is important to get medicines that show real promise to patients (for a reasonable price, however you define that) as quickly as possible, but not going into the area of “right to try”, which to me is the Wrong Way. So yes, I am full of contradictions!!!


  13. simpl says:

    The EMA didn’t cover everything in their one-page reply, though:
    – side-effects, particularly pain, may be a secondary aim in a study, but key for patients
    – cancer medicines of yesteryear , like phosphonates. relieved smptoms rather than closing in on underlying causes – even the idea of cancer stem sells seemed unnecessry 30 years ago
    – optimum usage takes decades to develop. Trials on late-stage patients may be too late for maximum effect.
    A friend with melanoma was offered either traditional repeated surgery or drugs, as nobody knew the interactions. I’ve not seen any discussion of antibodies backing up resection of a primary tumor, for instance.

  14. milkshaken says:

    Our CEO was fond of saying that cancer drug trials are so great for a small underfunded company because you can dose few dozens of patients for few months, see some response, maybe in 40% of patients, and unless you get too nasty side effects and somebody dies you can call it a great success… Now compare it with the length of clinical trial and required safety profile for a diabetes or cardiovascular, or multiple sclerosis drug

Comments are closed.