Skip to Content

Drug Development

Modeling the Rats, Who Model the Humans

When you get down to it, one of the biggest problems in drug discovery is that there is (in most cases) no alternative but doing things the hard way. If you want to find out if your drug is going to work for a given disease, there’s no other way to be sure than to give it to a bunch of people with the disease, under expensively controlled conditions, and watch them carefully (certainly for weeks, sometimes for years) to see what happens. Want to know if a new compound is going to be toxic? You certainly do – but the only way to do that is to give it to a bunch of animals, at varying doses and for varying periods. While that should get rid of a lot of bad actors, it still won’t eliminate the human-specific tox that might be lurking out there. For that, you still have to give it to people – or, if you’re really unlucky and there’s a bad but low-incidence effect, you get to wait until the drug’s actually been on the market for a year or two.

Even if you just want to know what a compound’s likely in vivo profile will be (ADME: absorption, distribution, metabolism, excretion), there’s only so far you can get without doing the animal experiments. In vitro liver assays (microsomes, hepatocytes) have helped, but there’s a lot more to ADME than the liver, important though it is. We’d really like to be able to sort compounds out an an earlier stage and sort out what the structural features are that help or hurt a given series of analogs, but there’s a limit to how many full pharmacokinetic workups a project can get done.

Enter computation and modeling. There have been many, many attempts at an <i>in silico</i> solution to this problem over the years, but it’s safe to say that none of them are solid enough to base real decisions on yet. Never once have I heard a project leader say “Hey folks, the ADME model rank-orders all the compounds like this, so we don’t have to worry about running these in rodents any time soon”.  Doesn’t happen – no one is going to draw any actionable conclusions until the animal numbers are in. The best you can do is with the M component. If your compound is shredded by liver microsomes, you probably do have a real problem, but good microsomal stability still doesn’t tell you anything much about its absorption, etc.

Here’s a new paper taking a crack at this problem, and it’ll be interesting to see how it’s received. The authors have a new computational approach (graph-based signatures), and to their credit, they’ve started a web server to let anyone try it out who wishes. (They note that no information is retained by said server, but that assurance, I have to say, is still not enough for anyone inside the industry to put any important structures into it – you can get fired for that kind of thing). What someone in industry can do, though, is put a bunch of already-disclosed structures through it, ones whose ADME behavior has already been characterized, to see how it does.

I’ll reserve judgment until I see some more data of this sort. The program uses the graph-based approach in addition to the traditional ones (physical properties, databases of structural features), and those latter ones are what we don’t trust enough already. So the key question is what the new algorithm adds to what we already have. The authors claim that it “achieved a performance as good as or better than similar methods currently available”, and specifically cite improvements in prediction of rat toxicity, P-glycoprotein inhibition, and inhibition of several CYP metabolic enzymes, among others. Of all the things on their list, the rat tox is to me the most interesting, because many of the others can be achieved by reasonably high-throughput assays, or (in the case of the Caco-2 permeability assay) are themselves not-always-reliable models of the real situation. A computational model of an in vitro model is not going to do anyone any good, but a reliable model of something like whole-animal tox could. We’ll see how this one performs!

48 comments on “Modeling the Rats, Who Model the Humans”

  1. xyz says:

    Derek, The font type on this blog is seriously giving he headaches; cant you atleast do something about it so that the commas and periods; colons and semi colons: dont appear similar?

    1. jsqwow says:

      I agree, it is not as pleasant on the eyes as the old font.

  2. bhip says:

    I appreciate these kind of efforts but what are the actions which will arise from the modelling data? Your statement-“no one is going to draw any actionable conclusions until the animal numbers are in” – pretty much sums it up.

  3. luysii says:

    Just to show how fraught extrapolation from rodent to man is, consider TDP43. This protein is of great interest to neurologists as mutations cause a variety of neurodegenerations. Even when NOT mutated, TDP43 accumulates in ALS, forming insoluble lumps in motor neurons.
    TDP43 is a 414 amino acid protein which binds to a variety of RNAs. A recent paper in Science (vol. 349 pp. 650 – 655 ’15 7 August) states that one can take the human TDP43 and have it function in worms (C. elegans), flies (Drosophila) and — mice.

    The function appears to be the inhibition of splicing of what are called cryptic exons, which if actually spliced in produce frameshift and early termination codons, leading effectively to loss of protein function when this happens.

    So why is extrapolation from rodent to man so hard? Because TDP43 suppresses cryptic exons in an entirely different set of proteins in man as opposed to mice.

  4. Recovering Mathematician says:

    I read this paper a while back. Initially I didn’t see how it got published, as the techniques used throughout are not novel, then I noticed the statistics and it suddenly looked more exciting. So I went to the experimental section in the supplementary information to see if I could knock up the models myself, when I noticed the sentence ‘after 10% of the outliers were removed’. Alarm bells started to ring the first time I saw it, by the time I’d read it a few times, I was pretty shocked. Further down in the supp you can read the validation sets used for these models, and out of the 30 different end points modelled, only five use a separate test set.

    This paper is an example of hiding poor practice in the supplementary materials and not being reviewed by the appropriate referees. Had this been sent to J. Chem. Inf. Model. I highly doubt it would have been accepted in this state.

    1. MolecularGeek says:

      The computational maven on the J.Med.Chem editorial board is Jurgen Bajorath. He’s been on the JCIM editorial board before, IIRC and he’s been around the block in modeling more than once. Until I have a chance to dissect it in more detail, I am willing to assume that the review the manuscript received was of comparable rigor to what a JCIM manuscript would have received. There are certainly other reasons why the authors might want to try for a JMC publication over JCIM.

      MG

  5. Andy says:

    So the purpose of modelling and other computer techniques in drug discovery is what?
    To employ computational chemists?

    1. Pete says:

      Hi Ash, I’ve seen variations on “Achieved a performance as good as or better than similar methods currently available” in other modelling articles and, strangely enough, numbers of parameters used to fit models never seems to get mentioned in connection with model quality. It seems that asking about such details is regarded as somewhat uncouth.

  6. Ash (Wavefunction) says:

    “Achieved a performance as good as or better than similar methods currently available”,

    I haven’t read the paper yet, but did their model work better than a null-model where you pick compounds at random? And did it give better correlations with tox than those with any number of very simple properties like molecular weight, logP, solvation energy etc. Those comparisons are important if you want to claim that your method works well.

    1. Chris says:

      I spent a while scouring the paper and the supplementary material looking for the details of the model to no avail. Surely the details should be made available so they can be replicated elsewhere?

  7. Pete says:

    As I infamously pointed out to a famous Austro-Hungarian medicinal chemist during (as opposed to after) his lecture, one important difference between rats and humans is that rats can’t vomit. On an unrelated note, there is a big difference to making models available on a web server and disclosing the models.

    As this is my first comment on IN THE PIPELINE in its new home, I’ll also say congratulations and may the blog be even more successful here.

  8. Bogus says:

    “ENTER computation and modeling?” Give me a break, it has been going on and delivering crap for years. And better algorithms are not the answer, either, because the bottleneck has always been, and always will be the amount of real data that you fit. There are just far too many variables than we can ever hope to get data for, so the signal will never break through the noise of overfitting that data.

  9. Anonzymus says:

    Bogus: Both better algorithms AND good data are the answer – it’s not an either/or game. I for one am very glad that someone is working on and publishing such methods. Even if they work poorly right now, the logical way to make them work in the future would be to invest more resources in them, not to dismiss them as irrelevant and stop working on them.

  10. johnnyboy says:

    Unless a computer model has the same level of sensitivity and specificity as a toxicologic assay, it will never be accepted as a substitute by any self-respecting toxicologist or regulatory agency. Even in the extremely unlikely possibility that this level of sophistication was reached, it would necessitate years and years of side by side testing before it was accepted. So what’s the point of developing these, apart from keeping computer geeks busy ? The only possible use I see would be as an indicator of potential toxicity done prior to the tox study (ie. “the model says there might be renal tox, so check the kidneys carefully”, that sort of thing) – not entirely useless, but certainly not worth a big resource investment. And as far as I know, such models are built on retrospective experience – such chemical structure has been shown to cause such toxicity, so similar structures may have the same, etc… But the whole point of doing animal studies is to check for novel, unanticipated toxicities, which show up all the time. No model would be able to detect these.

    1. Anonzymus says:

      Bogus: First of all, as models get better they get to a stage where they can make non-obvious predictions. This has not happened in the field of toxicology yet because the systems are very complex, but it has happened in other applications like pose prediction, airplane design and bridge design. Secondly, you are narrowing the potential applications of models to a very limited domain: The purpose of models is to not just make predictions but to make sense of data. It’s also to tell you what data to ignore. When you have a million toxicological data points even making sense of them and finding trends is beyond the capacity of human beings. That’s where modeling in the form of data analysis can help.

  11. Bogus says:

    Oh dear, you still don’t get it: Computer models can only tell you what you already know, because real data is the only source of truth and certainty in a model, besides fundamental laws of maths and physics. Anything else is just an extrapolation of that data based on arbitrary assumptions.

  12. Christophe Verlinde says:

    Most comments about pkCSM are quite unfair. This program was obviously not created to replace real world ADMET experiments. Instead the program aims at helping the medicinal chemist to navigate the vast chemical space out there in terms of predicted ADMET properties. If pkCSM helps me in reducing the number of compounds I have to synthesize with bad ADMET properties then it will have achieved its goal. The only fair question is whether it succeeds ON AVERAGE more often to send me CORRECTLY in the right ADMET space than it prevents me INCORRECTLY from exploring chemical space that I should have gone into.

    1. Pete says:

      I would challenge the assertion that “The only fair question is whether it succeeds ON AVERAGE more often to send me CORRECTLY in the right ADMET space than it prevents me INCORRECTLY from exploring chemical space that I should have gone into”. There is a misconception in drug discovery that any trend in data, however feeble, is necessarily useful.

      If models are to be useful in lead optimization then I believe that they need to be able to predict response of chemical/biological behaviour to structural changes. I also believe that SAR is essentially local and analyses of chemically diverse (heterogeneous) databases are of limited value to medicinal chemists charged with optimizing a specific structural series.

      One of the problems with ADMET models is that it is difficult to know how it applies to specific scenarios. There is always a concern about over-fitting when it is not made clear how many parameters have been used to fit the models. Can we make meaningful comparisons of models with different numbers of parameters that have been built and validated using different data sets? I’ve linked a blog post that raises some of these issues as the URL for this comment.

  13. Chris says:

    There needs to be better threading or nesting of the comments to work out who is replying to which comment.

  14. Anonymous says:

    Derek, Regarding your comment “…The authors have a new computational approach (graph-based signatures)…” about the article, a clarification is in order.

    The design of graph-based atom pairs signatures used in the article appears to be equivalent to topological atom pair fingerprints corresponding to all atom pairs in a molecule. The first mention of these fingerprints showed up in the literature almost 20 years ago. Like any other type of 2-D and 3-D fingerprints, atom pairs fingerprints do have their limitations.

  15. Anonzymus says:

    Bogus: First of all, as models get better they get to a stage where they can make non-obvious predictions. This has not happened in the field of toxicology yet but it has happened in other applications like pose prediction, airplane design and bridge design. Also, you are narrowing the potential applications of models to a very limited domain: The purpose of models is to not just make predictions but to make sense of data. When you have a million toxicological data points even making sense of them and finding trends is beyond the capacity of human beings. That’s where modeling in the form of data analysis can help.

  16. Bogus says:

    @Anonzymous: you still don’t get it. airplane design and bridge design are fairly mechanical and obey simple laws of maths and physics which are already known with certainty, so the extrapolations they make are based on these laws as robust assumptions. In contrast, biology is more complex with more variables (parameters) than you will ever get data points (observations). But just try solving any set of simultaneous equations where you have more variables (parameters) than equations (observations), and you will fail miserably. You will only end up with meaningless noise. This is not just my opinion, it is basic math/logic and information theory: You *can’t* make meaningful predictions when you will always have more unknown variables than observations. Period. Trust me. Would any other protein crystallographers and/or protein NMR folks out there vouch for me on this?

  17. Bogus says:

    Incidentally, the same goes for “Big Data”. There will always be more noise than signal, because there will always be more variables than observations. And as you look at more variables, the number of correlations you find will increase, but the proportion of those which are meaningless random non-reproducible statistical outcomes of pure chance will also increase. So you end up wasting more and more of your time looking at noise.

    When it comes to biology, Big Data and computational modelling are worse than useless, and always will be, due to the fundamental principles of information theory.

    Bottom line is: If you want to know something in biology, you just have to test it, because you will never be able to predict it unless you already understand the rules that govern it.

  18. Anonzymus says:

    Bogus: I am not implying that tox prediction is currently as easy as airplane design, what I am saying is that computers are good at detecting patterns that humans might miss because of the sheer amount of data, so there is no reason to believe that tox prediction will get better as both the algorithms and the data get better. Yes, some of the claims about Big Data are inflated, but that does not mean just not mean that we can do away with computational analysis of Big Data or that it’s totally useless. You are taking an extreme stand here. I disagree that we will never be able to apply modeling or Big Data to biology because of “fundamental limitations of information theory”. If that were the case then computers would never have been able to analyze the mass of genomic data that we are confronted with and computational biologists would be completely useless: try telling that to Francis Collins or to any large biotech company. We build predictive models based on incomplete data all the time, and the kind of overfitting that you are talking about is exactly what good modelers guard against. In addition there are statistical methods to estimate unknown parameters – people who do flux-balance analysis do this for instance. You are starting from a reasonable premise (“modeling is hard”) and going straight to a non-sequitur (“modeling is impossible”).

  19. Bogus says:

    In fact I agree with you that computers are great a spotting patterns in large and complex data sets. But what I’m saying is that’s exactly the problem, because more of those patterns will be meaningless, random non-reproducible artefacts, so we end up wasting more time chasing noise with computers than we would without. Now do you understand the issue, or would you rather wait until billions more are wasted chasing mirages in the desert?

  20. Bogus says:

    PS. The bottom line is that productivity of drug discovery has rapidly declined with all that genomic data, which only proves my point: At best it’s useless, and at worst (most likely) it’s a very expensive wild goose chase.

  21. Bogus says:

    PPS. In fact modelling is very easy, at least in biology, because you are fitting many more variables to relatively few observations (data points), but again that is precisely the issue, because the model you get is virtually useless for prediction.

    So modelling is easy in biology. But useful predictive modelling is virtually impossible. I have seen no model that consistently predicts what has not yet been directly tested. Have you? Or anyone?

    Well now you know why.

  22. Bogus says:

    OK, here’s one more attempt to convince you:

    If I asked you to join 2 data points with a 3-variable equation, I am sure you could do it very easily, but would your equation help you predict where the next data point will be? The simple answer is no, not at all, because with 3 variables your equation could be changed to cover every possible outcome while still being consistent with the first 2 data points.

    Similarly, if you have 10,000 data points and 10,001 variables, you cannot make any useful prediction. Not ever, unless you already understand the rules that govern the underlying biology.

    There, I’m done. 🙂

  23. SMILES2L says:

    @Bogus, you totally mixed up between math/physics and statistics/probability. The former pair is a real beauty, but the latter is more practical/useful in real life whenever it involves risks/rewards and data: (yes) in biology, chemistry, drug discovery, stock trading, and of course design of the plane, etc, Your examples are no more than what being taught in middle school. Keep a open mind.

  24. Chris says:

    Dear Bogus, I don’t normally respond to those who hide behind anonymity but I’ll make a one off exception for you.
    Medicinal chemists don’t make compounds randomly, we are always making hypotheses to explain the influence a new substituent has on activity and then using that very incomplete data to suggest new compounds to test the hypothesis.
    You suggestion that genomic data is useless would suggest you don’t believe the information about BRAC2 in cancer, the role of CCR5 in HIV, mutation in CFTR in cystic fibrosis is useful?
    As for your example with 10,001 variables, that is exactly the sort of the case where computational techniques can be useful. In all probability the majority of the variance will be explained by a relatively few variables, and that is where you need to focus your effort.

    And Derek whilst many of the models that are used to predict properties could be replaced by high capacity screens, the important difference is you don’t have to make the compound to test it in a model.

  25. tangent says:

    Mr. Bogus, discovering that the points in your high-dimensional space lie near a lower-dimensional structure is what science does. The number of point observations to identify that structure doesn’t depend on the dimensionality of the space it lives inside, it depends on the structure itself: the simplicity of the model. Which a priori nobody knows. (Even afterwards nobody knows the real truth.)

    Your intuition is that there is no model for a biological system that has any combination of utility and simplicity. Fine if you want to repeat that intuition very loudly, but realize that it’s your assumption, not a conclusion you can get by turning a crank on Algebra III.

  26. tangent says:

    (Ugh, stop the italics.

    The whole deal about simultaneous equations doesn’t make your case at all. You said “more variables (parameters) than you will ever get data points (observations)” which is a valid description (the only problem is if you knew the parameters you’d have a Nobel). Then you go to “equations where you have more variables (parameters) than equations (observations)”, but no, equational constraints are not observable, they’re the model you infer from observed data points.

    The argument you want to make is “you can’t characterize a model that has 100 degrees of freedom if you only have 99 data points to constrain it.” You could also go to “philosophically, inductive knowledge is impossible.”)

  27. Bogus says:

    Fine, OK, nobody is convinced by my theory/algebra/logic arguments. Then perhaps you will be convinced by reality:

    1. Productivity of drug discovery is declining as fast as it always has been despite all the genomic data and computational power.
    2. There are still no models that can predict whether a drug will be safe and effective, even with the remotest level of statistical confidence.

    Ergo, models are at best a useless distraction, and at worst downright misleading and expensive.

    Now let’s see if you can argue with reality, or if you still have your head in the sand (which is effectively looking with models)

    And regarding certain markers like BRCA, yes, some of what data analysis finds turns out to be right, but far too much of it is crap to make economic sense.

  28. Bogus says:

    PS. And the fact that success rates are decreasing in every phase of development only confirms that we are spending more and more time looking at noise rather than signal. And that is exactly what you would expect when you keep mining more and more data: eventually you end up sucking sand.

    Has nobody here read any books by Nate Silver?

  29. Bogus says:

    @tangent: Correct, inductive knowledge is impossible, unless you already understand the underlying rules and logic, which are effectively constraints between variables, much like the data points themselves.

    The only thing you can induce when you have more degrees of freedom than data points and logic/other constraints, is a hypothesis. Which is why you still need to test everything anyway.

    The problem is that hypotheses become weaker (more noise) as you look at more variables (e.g., specific mutations) than people we have available on the planet to sequence and observe.

  30. LeeH says:

    A few comments.

    First, it is fundamentally untrue that you cannot build a predictive model where you have more variables than cases. It’s true that it’s more difficult, however there modern machine learning methods such as Random Forests that are inherently resistant to wide data, and variations of others that are similarly useful. Anyone who does not believe this assertion should look at the data mining literature, where this is essentially a solved problem.

    Second, there seems to be a misconception about the usefulness of an imperfect model. To quote Einstein “all models are wrong, some are useful”. This is very true in drug discovery. The judicious application of models can demonstrably bias, for many properties, the success of a program. Note that this can only be measured over some reasonable number of compounds, in order to see the statistical advantage. Any application to a single compound is nonsense.

    There also seems to be a misconception on how to use these kinds of models. To my mind, once the compound is made the model is useless. Models are useful for biasing the selection of compounds to make, not to replace an experimental determination, because you can’t really afford any uncertainties (beyond the experimental ones).

  31. Bogus says:

    @LeeH: “The judicious application of models can demonstrably bias, for many properties, the success of a program… Models are useful for biasing the selection of compounds to make”

    So why are success rates falling so fast, when models are supposed to be getting better all the time?

  32. LeeH says:

    Bogus:

    Models are not a panacea. They give you a statistical edge when deciding on what compounds should be made, making it somewhat easier to find an advanced lead. So it gets to into development somewhat faster, but you still are vulnerable to the uncertainties of testing for ADMET properties in animals, which as you probably know are often not good surrogates for humans.

    And on ADMET, the models in this area are, relative to those for target-based problems, still very weak. If you think about PK, for instance, there are myriad factors that go into the effect itself. Modeling excretion, for one, is damn near impossible. Absorption is much better understood and predicted, especially in the age of Lipinski and derivative rules, which do a pretty good job (but just for this one property!).

    Finally, to expect that models can overcome the increasing complexity of finding a drug in today’s world, I believe, is unfair. The added burdens of ever-increasing safety requirements, coupled with competition from effective and cheap generics, is way too much to compensate for with one set of tools in our tool belts.

  33. Bogus says:

    @LeeH: “Finally, to expect that models can overcome the increasing complexity of finding a drug in today’s world, I believe, is unfair.”

    Why is it unfair to expect modelling to overcome the increasing complexity of finding a drug? Isn’t that *exactly* what we are all trying to do here, and exactly what you said modelling is (should be) useful for:

    “The judicious application of models can demonstrably bias, for many properties, the success of a program… Models are useful for biasing the selection of compounds to make”

    Either modelling helps, or it doesn’t. And the declining success rates suggest that it doesn’t. And now you are suggesting that it can’t. So why on earth are we doing it???

  34. LeeH says:

    Bogus

    Modeling is not magic.

    You seem to be making the case that a method is only useful if it overcomes all the deficiencies in all of the other parts of the drug discovery process, turning a declining success rate into an improving one, essentially . Using these criteria, everything that is done in Pharma to increase productivity is of no use, since industry drug output is down. This is a specious argument.

  35. Bogus says:

    @LeeH: “Using these criteria, everything that is done in Pharma to increase productivity is of no use, since industry drug output is down.”

    I couldn’t have said it better myself.

    Maybe a completely different approach is required. Because the industry won’t last much longer on its current trajectory, with or without modeling.

    BTW, thanks for the discussion. I tend to be provocative, but I mean well. 😉

  36. Soo says:

    Now, if we only would apply the same critical rigor towards climate modeling, which has greater complexity and far less opportunity for laboratory validation…

  37. Bogus says:

    @Soo: It would probably be easier (and more productive) to model the impact of human behaviour on changing the legislation as required to manage the climate, as at least that may come up with some ideas for change that people are willing to accept…

  38. LeeH says:

    Ah, Bogus, I’m not sure you understand what specious is…

  39. Bogus says:

    @LeeH: I do, but I haven’t seen a more truthful argument that fits the facts. 😉

  40. Ash (Wavefunction) says:

    Bogus, what do you think of the recent FEP paper which Derek blogged about? There were demonstrable successes in that paper: for instance 71% of the compounds predicted to have higher potency actually did while only 7% of the ones that were predicted to be inactive were active. Good luck doing that kind of a prediction manually. You cannot set up a straw man by expecting modeling by itself to solve all kinds of attrition in the industry and then spend seven comments knocking it down. No tool – computational or experimental – can do that. That’s why we work together instead of trashing each other’s fields – perhaps you would rather us not…

  41. Bogus says:

    @Ash, I think the paper is interesting and even encouraging, but with lots of caveats, including publication bias as I wouldn’t expect to see any papers reporting negative results.

    But still it all comes down to productivity, and right now (even as we all work together), there is just not enough to sustain an industry. So I would rather be frank about that and try to do something about it by trying a completely different approach, than pat each other on the back with our heads in the sand, doing what we’ve always done and pretending all will be fine. But that’s just me.

  42. Ronke says:

    I think the point Bogus is making an economic one, arguing that despite modeling productivity has not improve while others are making fine philosophical points about our quest for knowledge, that modeling may not answer all our immediate questions but there’s a method in the madness. All well and good. To placate Bogus’ productivity issue, the questions then are; is modeling responsible for the low productivity? Is modeling employed solely to solve productivity issue? Has modeling been helpful at all? Why is there low productivity in the first place?

Comments are closed.