Skip to main content
Menu

Life in the Drug Labs

The Killer Experiment

Bruce Booth has some thoughts here on a recent Harvard Business Review piece on startups, but don’t let the fact that it’s from HBR put you off from taking a look. The original article is focused on innovation in general, but Booth ties it more directly to biopharma culture, and his advice certainly looks sound to me. And it’s not just sound for people working in small companies, although they should definitely read it. The same lessons apply to doing R&D in any size organization.

I wanted to highlight one of those: the willingness to design and to run “killer experiments”. And by that I don’t mean in the “whoa dude, killer experimental design” sense, but rather in the “hard decision point” sense. Not every phase of one’s work lends itself to these things, but you will have opportunities to set up stuff that’s pretty close to pass/fail, and (here’s the hard part) you should not miss the opportunities to do so. Now, I’m not suggesting that you should wander around all the time trying to find excuses to kill your project. Every drug project has plenty of reasons why it can be killed. What I’m saying is to be alert to the chances to run definitive experiments that will put your hypotheses to  a solid test.

Designing these experiments shouldn’t necessarily be easy – you’ll want to think hard about the best way to put things to the most direct test you can, with enough well-run controls that you’ll believe the result even if it’s bad news. That brings up something I’ve mentioned before, and that many readers will have experienced: sitting in a project meeting where a particular experiment is being proposed, and thinking that yes, if the results are positive, you’ll go on with the plan. But if the results are negative. . .well, you know, when you put it that way, you’ll go on with the plan anyway. At that point an alarm bell should go off to make you ask why you’re running the experiment anyway. It’s not that these are always useless, of course (sometimes you have to collect a value for some parameter, just to have it), but sometimes they are. The most insidious are the experiments that get billed as decision points when in reality they’re no such thing.

A real decision point experiment should make you nervous, frankly. You’re putting a lot on the line, and you should be committed to being ready to kill something off (some compound series, some approach, some assay, some hypothesis, maybe even some entire project) if things don’t go right. Don’t cheat. If you’re not going to follow through on what the experiment tells you as designed, redesign the darn thing to make it more convincing. Otherwise you’re just fooling yourself and/or trying to fool somebody else, and neither of those is an honorable or productive use of your time.

A corollary to all this is that such experiments should be run as early as it’s feasible to run them. The longer you wait, the more you risk (time, money, energy, and so on) and the worse will be the temptation to carry on no matter what. The normal tendency is to look for excuses, for reasons not to do this sort of thing, to fill in your busy schedule with all that other stuff instead of facing the music. Don’t. It’s bad enough that we work in a business where the killer experiments (clinical trials!) often have to take place after a great deal of time and money has been spent, but you don’t have to add to that in your own part of the process. Think about the most likely things would invalidate your work, and what the most solid, undodgeable bad news would look like. Then go make sure that it’s not there – do your best to conjure up those demons, because if you don’t do it now, you’ll be forced to do it later.

This is contrary to normal human psychology, but then again, the whole scientific method often seems to be contrary to normal human psychology.We’re wired more to fall in love with our own ideas and to pick and choose the evidence we will accept for and against them based on how it makes us feel. But that’s asking for disaster in science. We’re up against physical laws and phenomena that could not possibly be paying less heed to how we feel.

So put your ideas to the test. Take them up to the mountain and hold the knife over them. It’s the only way.

42 comments on “The Killer Experiment”

  1. analoger says:

    what?? you mean like the time I made as many ridiculously diverse analogs of a compound as I could to prove there was no “specific” interaction going on?

    Needless to say I was persona non gratis after that, good way to figure out if you’ve signed onto a pump and dump tho

  2. Wavefunction says:

    The biggest problem with not running killer experiments in my experience has been management mindset, which in large part comes from the sunk cost fallacy and plain wishful thinking: once a project has been going long enough, management is reluctant to run a killer experiment – even a relatively easy one – because of wishful thinking, hubris and emotional investment.

    There are a lot of analogies from the tech world which I don’t think should be applied to the biopharma world, but I do think that the principle of “fail early and break things” should be applied much more widely to experimental drug design.

    1. Snakeoilvendor says:

      And the number one reason for this???
      Stock price….. Or an impending IPO
      Few people get the chance to work in enough of these places to realize what’s really going on in the minds of the founders and VCs
      Most are just thankful for a paycheck,and could care less

      1. broken knee says:

        Agreed.
        Who pays the piper picks the tune. Even if they have no music knowledge.
        Scientists beware.

    2. NJBiologist says:

      Ash, I can’t remember if it was in the HBR article or the commentary on Twitter that led me to the original, but there was an emphasis on doing the killer experiment *as soon as possible* to mitigate the sunk cost issue. (That doesn’t get rid of it, but it has to help.)

      1. Derek Lowe says:

        And the tricky part is, in drug discovery “as soon as possible” may well still not be enough to save you from spending a lot of time and money, especially if you’re breaking new ground. But the principle stands.

    3. eub says:

      Tech companies are not real good at actually doing this, specifically the part where you decide “no, let’s not do this.”

      More specifically, they are (in my experience) good at saying nope to experiments, even large ones, but bad at saying it to projects. This may seem like a weirdly fluffy conceptual thing, but it matters.

      The organizational distinction is that one team sticks around and runs multiple experiments. Some go forward, some don’t. They are fine with killing the ones that don’t seem to work. But when a team was formed for the purpose of a project, that’s very different. If it dies, they, well obviously they don’t die in the literal sense, but they suffer upheaval and may be at risk for being reassigned or even laid off. (My industry doesn’t do layoffs here but the upheaval and emotional sense of We Lose is a really big deal that affects behavior.)

  3. Hap says:

    It’s better to kill an idea with an experiment rather than an assumption.

    1. RICHARD BERNSTEIN says:

      Better to kill and idea than a patient.

      1. A successful drug is one that reduces the number of patients suffering from the symptoms?

        1. James Wheatley says:

          And thus the novel approach of concentrated HCN as a cure for asthma was born…

  4. Pay Up $ucker says:

    Wrong. It’s unethical to kill this project when my job and the jobs of the 50 other people at the company depend on this project moving forward until some dumba$$ Big Company comes along to pay us 10X what we’re worth…

    1. Desperatechemist says:

      And how long till people get wise?

      Primary reason for tens of thousands of chemists losing their jobs in the R&D downturn, without a doubt

      1. Pay Up $ucker says:

        This has been going on for years. I was at a Big Company that told us to stop doing experiments in an area that they were going to do a deal in, because they didn’t want to know that it wouldn’t work.

        There are.no signs that people will.get wise any time soon.

        1. Urcutoff says:

          The lack of investment dollars by VC’s for anything small molecule, or any company not at “clinical stage”….is a clear indicator

  5. John Wayne says:

    Derek, shockingly poetic biblical reference. Well done.

    1. Derek Lowe says:

      Hit me just as I was writing the end of the post – probably a flashback of Kierkegaard as much as the Old Testament.

    2. sgcox says:

      Second to that. Alas, subjective truths >> objective truths

  6. Anonymous says:

    I like to think that my undergrad research experiences and mentors taught me many valuable (and correct) lessons about scientific research but, unfortunately, did not prepare me for grad school org synth or biotech or beyond. Early on, I had read “Advice to a Young Scientist” by Peter B. Medawar (Nobel Prize) and he spoke of “Il Grand Cimento” – The Grand Experiment – or killer experiment (as described here). Design it well and do not be afraid to conduct it. That’s what I learned as an undergrad. “Do it or merely suggest it and get canned” is what I learned thereafter.

  7. azetidine says:

    If you can kill your project (for a good, scientific reason), then you should kill your project.

  8. Calvin says:

    I think most people agree with this whole approach, especially when it’s somebody else’s project ( 😉 ). That’s the human behavior part.

    The only “problem” with this approach is that it presupposes that the output of the killer experiment is a binary yes/no or provides clarity. The real issue is that when, all too frequently , the output of the experiment is a “meh” or better yet a “what the hell…..?”. And at that point you have no idea if the result is a killer, or if the experiment itself should be killed. I think that happens all too often and that’s when human behavior drives decisions for better or worse.

    For example, I remember once putting my best compound through a transgenic mouse model of a particular infection. Yes my compound worked, but not as well as expected and with apparent signed of fast onset resistance. We’d had the good fortune to have a positive control with human clinical data too, so that went in and gave slightly weird results as well. It was definitely a critical experiment, and we thought of it as a killer experiment. But all we got was a lot more questions (we never used that model again and resolved that going into patients was the only way so maybe it was useful) and a “resistance” stain attached to the series when all the other data suggested the opposite (the resistance also could not be identified and there were other very weird virology issues).

    So yes, do the killer experiment but I think a lot of judgement (which is open to gaming in all directions) is going to be required when the experiment gives that “meh” result. Or just get to patients asap. And at that point for sure do your best not to delude yourself. Good luck

  9. “We’re in a risky business, and we hope if anything happens to us, it will not delay the program. The conquest of space is worth the risk of life.” — Gus Grissom, commander, Apollo 1.

    Oh, wait, you didn’t mean that sort of killer experiment? Never mind.

  10. Peter Kenny says:

    I’ve always thought that term ‘euthanize’ should be used more enthusiastically by project mangers (or leaders as they would prefer to be called).

    1. Levorotatory says:

      Never judge a man until you have walked a mile in his moccasins

      1. Mark says:

        That way, when you do judge him, you’re a mile away! Also, you’ve got his moccasins.

        1. Peter Kenny says:

          The logic is flawless although I would prefer first to conduct a killer experiment to determine whether the man has athlete’s foot

  11. Barry says:

    At the center of Drug Discovery is Research. At the center of Research is Science. At the center of Science is the effort to falsify your hypothesis. Almost very drug discovery effort has an hypothesis of the form “modulation of this biological target will improve outcomes of this disease”.
    The experiment that could falsify that hypothesis could free vast resources for other projects. But no one ever got that promotion for such an experiment; that reward is reserved for bulling forward

  12. JB says:

    Hasn’t fail fast and fail early been the dogma for decades now in biopharma? What’s new here? They just reworded it as ‘killer experiments’.

    1. Skeptic says:

      Yes, but it bears repeating because of human psychology *and* the usual incentives. For almost everyone but a long-term investor, the incentives promote the accumulation of good news, not bad. Bench scientists don’t want to risk their job or that of their colleagues. Managers don’t want to risk their funding or their project. VCs don’t want to risk the chance of a lucrative IPO sooner rather than later.

      To want to fail early you have to be in a position to benefit from the success of the next project after the current one is killed. Most people involved aren’t.

  13. hn says:

    Many academic labs are one-trick ponies. PIs will do anything to protect their core project.

  14. TroyBoy says:

    It’s easy to be glib about doing the killer experiment and going with the result.

    Three counterpoints:
    One, our biological models are imperfect. What preclinical biological model/assay perfectly correlates with clinical outcome? Cell lines and xenografts all have their problems. Should we be willing to kill a program on an experiment when we don’t understand the biology of the cell line or mouse model well enough to make a conclusion? So my question is: what *is* a killer experiment?

    Two, killer experiments are difficult for innovative first-in-class/new mechanism of action programs. What if your drug target has never been targeted before and there are no comparator molecules? It’s easy to do the comparator experiment if you’re making the 19th checkpoint inhibitor. So should the field only stick with programs where we drug the same types of targets?

    Third, I’m going to bring up a quote from a previous post of Derek’s:
    While it’s easy to point to expensive failures and criticize organizations for not pulling the plug sooner, it’s also true that just about every successful drug faced some legitimate existential crisis along the way — at some point during its development , there was a plausible reason to kill the program, and someone had to fight like hell to keep it going.

    How many programs have been killed because they failed the “killer” experiment? We will never know. But the thought of killing a legitimate program early should scare us as much as not killing a program until you get the Phase 3 results. I guess what I’m trying to say is that we need to take results in the aggregate. Drug discovery is a multi-dimensional process of optimization with multiple criteria. Relying on the result of one assay or model to do a killer experiment is unwise. Many successful marketed drugs may have not performed well in a key “killer” experiment, but a champion had to resurrect it.

    1. Failed Alot says:

      TroyBoy,
      Fair questions – let me try to add additional perspective.

      1. Nobody says that designing the killer study is easy. Or even possible. You need to have the right tools, and you don’t always have them. But sometimes you do. And that happens when you spend time and effort creating them. And that’s the time the drug-hunter goes for the killer shot.

      2. Good question; my answer is no; but you cannot run a project for a novel target the same way you run a project for a better-known target. As I see it, most drug discovery projects these days go for novel biological targets or mechanisms – much less interest in me too drugs. So, back to point 1: you work to create the knowledge and tools to run the killer study.

      3. Correct. By the time you get to clinical development a lot of money has been invested, and it is hard to make these go/nogo decisions. In early drug discovery, those pressures are less so. Given the large number of early activities, organizations that have the discipline to conduct killer studies when possible will have a much more efficient early pipeline, IMHO.

      Sometimes the killer study is just doing good science. For example, don’t use your 1 nM “selective” kinase inhibitor at a 100 micromolar concentration, where it is inhibiting every possible kinase and their brothers. It worked at 100 mM… good, what happens when you reduce its concentration? A simple study, just takes guts to face the possibility our hypothesis is not correct.

      Last time I checked we all wanted to do science in an hypothesis driven way. So, let’s put the money where our mouth is: create a clean biological hypothesis and build the tools to disprove it. Yes, things are never black or white in drug discovery. Biology is too complex for my post-Neanderthal brain. But sometimes things are brown and smell like crap. At least then, we can make a decision.

      Bottom line: good judgement comes from experience, and experience comes from bad judgements. If you fire all your experienced colleagues and want to succeed at drug discovery, you need to be ready to pay for all the bad calls to be repeated.

      1. TroyBoy says:

        Failed Alot, I appreciate your comments. You make some very good points. And I agree that we all want to do good hypothesis-driven science. You have to be willing to follow the science. Handicapping people that want to do well-controlled critical/key experiments is unwise.

    2. Anonymous says:

      Some killer experiments are fairly easy, such as, RUN THE PROPER CONTROL!

      I’ve got a story about the new hire who came in and ran the “standard” assay. It was a coupled enzyme system. However, she set it up to ALSO run the compounds against the enzymes independently and found that many inhibited everything, not just the target enzyme (as had been “assumed” but not tested). They had to go back and redo several man-years of assays. Some hot leads became non-selective dead ends.

      “Il grand cimento” in organic synthesis happens every now and then but is a little more subjective. Organic Syntheses is a kind of pre-pub check on reproducibility. When 90 MHz NMR was routine and you needed permission to use the 400 (1980s), I recall a PI who refused to allow some samples to be NMRed on the 400 … because it would have exposed (killed) what we already knew: the sample was a mixture of regioisomers, indistinguishable at 90 MHz and the reaction was not selective. Sometimes, a remarkable result just has to be reproduced by another researcher and chemistry is full of examples of frauds that were exposed (killed) AFTER the fact (asymmetric synthesis in a magnetic field; Sames-Sezen; etc.).

      Proof that something does work (a “life or survival experiment” cf. “killer experiment”) is often more difficult and Williams’ reproduction of Rabe’s synthesis of quinine from quinitoxine (in support of RBW’s 1944 relay synthesis) took A LOT of effort.

  15. Sad but true says:

    When i advised my millennial colleague to design the killer study they complained to HR and i lost my job. Too disruptive.
    The better advice in that article is “never accept incompetence”. Competent people are not afraid to run the right study; they would hate to waste their time.

    1. eub says:

      Hoo boy that’s one of those stories that screams “selective narration” all right.

      1. anon3 says:

        Maybe the ‘killer study’ was some kind of sexual advance?

        1. Sad but true says:

          … well, the project was on PDE5 inhibitors and how to differentiate them from existing marketed drugs…. what does that killer study look like? 👀

          Now, seriously, isn’t it the point that during times of record investment in biotech, Bruce Booth feels the need to write this article on basic cultural elements of successful startups? If everything is working so well, why the article?

          1. eub says:

            Aw, you’re changing the subject from precisely how your advice to a incompetence-accepting millennial colleague became a firin’ HR complaint. You’ve got an eager audience for that one! Lay it on us!

  16. Longbin says:

    No contest to the concept but completely agree with Calvin and Triyboy

  17. me says:

    Strangely I have an example of quite the opposite: we had killed a project off and I devised a ‘killer experiment’ to kill the killer. After a shouting match with my manager, he agreed to run it (quite expensive test on a whole raft of compounds we’d abandoned). You can imagine how he presented the result to the senior management team when it turned out I was right.

  18. Joe Q. says:

    We used to call it “do the last experiment first”.

    The problem is not necessarily with designing and running the “killer experiment”, but — as explained earlier — with figuring out what to do if the result is inconclusive.

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.