Skip to main content

Drug Assays

Phenotypic Screening For the Win

Here’s another new article in Nature Reviews Drug Discovery that (for once) isn’t titled something like “The Productivity Crisis in Drug Research: Hire Us And We’ll Consult Your Problems Away”. This one is a look back at where drugs have come from.
Looking over drug approvals (259 of them) between 1999 and 2008, the authors find that phenotypic screens account for a surprising number of the winners. (For those not in the business, a phenotypic screen is one where you give compounds to some cell- or animal-based assay and look for effects. That’s in contrast to the target-based approach, where you identify some sort of target as being likely important in a given disease state and set out to find a molecule to affect it. Phenotypic screens were the only kinds around in the old days (before, say, the mid-1970s or thereabouts), but they’ve been making a comeback – see below!)
Out of the 259 approvals, there were 75 first-in-class drugs and 164 followers (the rest were imaging agents and the like). 100 of the total were discovered using target-based approaches, 58 through phenotypic approaches, and 18 through modifying natural substances. There were also 56 biologics, which were all assigned to the target-based category. But out of the first-in-class small molecules, 28 of them could be assigned to phenotypic assays and only 17 to target-based approaches. Considering how strongly tilted the industry has been toward target-based drug discovery, that’s really disproportionate. CNS and infectious disease were the therapeutic areas that benefited the most from phenotypic screening, which makes sense. We really don’t understand the targets and mechanisms in the former, and the latter provide what are probably the most straightforward and meaningful phenotypic assays in the whole business. The authors’ conclusion:

(this) leads us to propose that a focus on target-based drug discovery, without accounting sufficiently for the MMOA (molecular mechanism of action) of small-molecule first-in-class medicines, could be a technical reason contributing to high attrition rates. Our reasoning for this proposal is that the MMOA is a key factor for the success of all approaches, but is addressed in different ways and at different points in the various approaches. . .
. . .The increased reliance on hypothesis-driven target-based approaches in drug discovery has coincided with the sequencing of the human genome and an apparent belief by some that every target can provide the basis for a drug. As such, research across the pharmaceutical industry as well as academic institutions has increasingly focused on targets, arguably at the expense of the development of preclinical assays that translate more effectively into clinical effects in patients with a specific disease.

I have to say, I agree (and have said so here on the blog before). It’s good to see some numbers put to that belief, though. This, in fact, was the reason why I thought that the NIH funding for translational research might be partly spent on new phenotypic approaches. Will we look back on the late 20th century/early 21st as a target-based detour in drug discovery?

36 comments on “Phenotypic Screening For the Win”

  1. Barry says:

    various groups have reported over the last decade that they could make e.g. blockers of B-RAF that were potent in vitro, yet only some of them blocked signalling as measured downstream at MEK or ERK in whole-cell assays. It was years later that this was explained by lateral signalling through BRAF/C-RAF heterodimers.
    Still, these compounds that couldn’t evoke a signal in a whole-cell assay died in the Research phase, not in the Clinicals. To leap from the binding assay to the Clinic without seeing cell-based activity and Pharmacodynamic response in whole animals would be to practice Faith-Based Research in lieu of Science.

  2. lynn says:

    Phenotypic musings: In the antibacterial area, phenotypic screening is not just looking for killing of bacteria [that would be ’empiric screening’] – but for evidence that a specific target or pathway has been affected. Phenotypic screening is analogous to looking for genetic mutations by screening for the whole-cell phenotype they should engender. Similarly, when looking for inhibitors of specific intracellular targets, knowing the phenotype of a conditional mutation (under semi- or non-permissive conditions) gives a model readout for a screen. In this regard, having conditional mutations in a potential target gene is more useful than having knock-outs – since you can observe the effects (temporally) of shutting down the activity, which can be similar to the effects of an inhibitor.

  3. MTK says:

    Hhhmmm. Honestly, not my field of expertise, but let me ask a couple of questions regarding this. And oh yeah, just to complete my ignorance I haven’t read the paper either, but…
    a) Is 28 vs. 17 really significant? Would that not depend on how many phenotypic screens vs. target screens are conducted?
    b) Are we not missing a denominator or two here? Besides # of screens don’t we also need to know avg cost in dollars and time? I guess what I’m saying is that the data needs to include something about productivity not just absolute numbers, right?

  4. RickW says:

    Other than the initial choice of strategy, is development of new screening assays a bottleneck in phenotypic approaches?

  5. Virgil says:

    Recarding average costs, the general consensus is that phenotypic screens are more expensive. Taking an Elisa plate out of the freezer and adding your compounds, is way cheaper than growing up cultured cells, transfecting them, and assaying a biological endpoint. Hence the focus on the former.
    Biological systems tend to be more “picky” about conditions, e.g. temperature control. You could run things at 25C, but you’re better off at 37C, and maintaining temp’ control on liquid handling robots etc. can be tricky. Light is another one – cells behave very differently in light vs. dark conditions.
    The other big downside to pheno-screens is specificity. It’s easy to be specific in a ligand-binding Elisa because there’s pretty much only one thing in the pot that will give you a signal. When dealing with receptors on living cells, and downstream outcomes (e.g. cell death), the number of things that can give you a false positive or false negative increases exponentially. More controls are required = more money and time.
    In addition to the greater expense of actually designing and running the phenotypic assay (see above), phenotypic screens are also less amenable for multiplexing (i.e. testing lots of drugs all at once in a single well). For example, if screening for ligand binding to a receptor, you can throw 10 drugs in the pot and the most potent binder will win out. If you throw the same 10 drugs at a cell, good luck interpreting the outcome! Thus, at 1-drug-per-well the throughput is lower, which affects the expense.

  6. MoMo says:

    Let’s hope target based discovery is in its death-spiral, as we all start to figure out that drugs have multiple targets and the one touted necessarily is’nt the only one affected or operable.

  7. MTK says:

    Thanks for the info, Virgil. I sort of figured that would be the case.
    I guess I was also trying to get at the cost of compounds coming out of phenotype vs target based screens in terms of clinical failures rates, also. A priori I’m not sure that there should be a difference since they both have to jump through the same pre-clinical hoops, which is sort of why I asked question a: Is 28 vs. 17 significant or just chance?
    The actual screens themselves can’t cost much in the big scheme of things, but the consequences afterwards may or may not. It seems you have to ask a series of questions to try and get an answer as to which approach is better.

  8. Screener says:

    The key for the eventual (clinical) success of phenotypic screens vs. target-based screens is building cell-based assays that are relevant to the human disease. Here is where we come back to the deeper understanding of basic biology being the area where more needs to be done. That is the bottleneck my friends.

  9. anon says:

    For a compound to become a successful, useful, effective drug must be “about the biology” — e.g. the impact it has on the phenotype (however defined). Characterization on biology / phenotype must be done, no matter if the chemistry is said to be “driven” by other data, such as target based screens. The latter is just another tool in drug research’s arsenal, in the hope of making the overall job easier, faster, cheaper, focussed, and should not be taken as an end-game unto itself. When this happens, the effort typically fails since “potency on target” does not itself a drug make.

  10. Anonymous says:

    This study pretty much states the obvious, at least to us guys who were doing drug discovery in the 60s and 70s.
    However as a chemist, I find the reductionist molecular approach to drug discovery so very seductive and intellectually satisfying. It will be damn tough to let it go.

  11. S says:

    In a sense, target based screens and phenotypic screens are not that different – they just use models of varying complexity. More complex models (whole organism/phenotypic) are potentially more accurate, but not necessarily so.
    The results are as good as the model, and no better.

  12. Rick says:

    S, #11,
    I’m sorry, but target based screens and phenotypic screens are literally incalculably different, especially when it comes to target-based in vitro enzyme screens. For starters, look up the free water concentration inside your average cell and compare it to the free water concentration in an in vitro assay.
    Secondly, phenotypic assays typically assess multiple potential targets at one time (at least if you’re not asleep while you’re doing them), whereas target-based screens only assess, well, one target or if they’re a “really smart” screen, multiple targets at one time, but still nowhere close (as in not within a couple of orders of magnitude) the number of potential targets sampled in one phenotypic assay.
    One of the biggest problems with target based assays is that people overestimated what they could sample and underestimated the difference with phenotypic assays.

  13. Rick says:

    In answer to your final question, “Will we look back on the late 20th century/early 21st as a target-based detour in drug discovery?”, I must say I am skeptical if by “we”, you mean most people associated with the pharmaceutical industry and by “look back” you’re thinking of any time in the next decade.
    Remember that, without exception, every presentation of “THE DRUG DISCOVERY PROCESS” starts with “Target Selection”, then “Target Validation”. We have a generation of drug discovery scientists and managers who have been taught (and I have heard this for myself) that you cannot discover a drug if you do not know your target. I agree with you that this myopic, historically inaccurate view of drug discovery has had a demonstrably crippling effect on drug discovery.
    The opportunity cost of NOT exploring an area of research because it wasn’t borne of the “new paradigm” of drug discovery, which required picking a “validated” molecular target, is incalculable. Worse yet, the number of scientists with experience at drug hunting without being handed a clone of a validated target is rapidly dwindling, because they were thrown out of the business as dinosaurs not hip enough to participate in “modern drug discovery”. Now they’re too discouraged, retired or dead to try to teach those who would not believe for so long.
    I was a bit encouraged when the term “phenotypic screening” came along (despite being a bit chagrined that it was nothing more than good, old fashioned pharmacology dressed up for the 90s, but that attempt at putting new wine in old skins didn’t seem to impress investors. Then along came chemical genomics and chemical biology – again, fancy new names for “pharmacology” – and a few companies tried to build on that, but didn’t really get enough support for that to make last long in that vein. Maybe someone will come along with another new name for the same old thing, but soon everyone will realize it’s just pharmacology again and that was discarded as too old fashioned in the 80s because we just got so damned smart. Yeah, right.

  14. cliffintokyo says:

    I also favor the holistic approach to discovery. The in vitro, in vivo, whole cell, animal pharmacol, metabolic studies, toxicology, etc should ideally all contribute synergistically to the growing understanding of the profile of hit compounds.
    I have worked with pharmacologists for whom it was anathema to suggest that the hit compounds might be hitting multiple targets, and that it would be useful to investigate.
    Its also possible that these people had been disincentivized by the myopic financial overseers at the company in question.
    Budget driven HTS silo mentality among biology staff is deadly for pharma industry research.
    On a different tack, I once flabbergasted my boss by suggesting that we should use natural products that are known to be human poisons at low doses as potential leads.
    “At least they have potent biological activity”
    I said!
    We might discover some new leads by investigating the MMOA of such compounds?

  15. imatter says:

    When the term “phenotypic screening” came out, I initially said to myself that it’s also called pharmacology and sometimes medicninal chemsitry.
    It think the problem with targeted screening is that often forget a cell is not just ON/OFF system. Sometimes, a cell is complex enough to respond to survive an assault by turning on an alternate pathway to survive. Often, a target protein is redundant.
    Still, both approaches have their appropriate use and role drug discovery.

  16. Cellbio says:

    The specificity issue is really no different than target based specificity issues. Yes, if all someone does is screen one biological system for activity, then you have a specificity issue, but the same would be true if a kinase panel were forgone for screening only the intended target. When I ran a phenotypic screen in cell assays, we made more than 20 measures of biological impact and instantly could assess biological specificity, and had no problem identifying compounds (and SAR) with more restricted biological impact than drugs with known mechanisms.
    In contrast, once target-based leads passed potency and biochemical screens, they almost never went into biological specificity screens, that is until early tox studies. We started to assess target-based molecules in our “system”, and revealed that biological specificity revealed a lot more pharmacological distinction amongst compounds that appeared no different in known biochemical profiles. We even saw a compound that obliterated all measured biology, but that had also met every hurdle to make it to scale up and tox in a target-based program.
    I agree with imatter that this is just pharmacology, but was previously called a dinosaur for not flowering the effort with novel linguistics. Seamless Execution of Chemical Biology Nodal Network Screening in a Multiplexed High Content Target-Native Metabolically Competent Setting?

  17. Mark says:

    If you look at some of the really transformative drugs developed (i.e. benzodiazepines), they were identified by phenotypic screening.
    If I remember correctly, bdzs were originally synthesized in an attempt to make a new antibacterial and the sedative effects weren’t recognized until the drugs were dosed in mice/rats. What if they had been relying on a target screen? Or even on bacterial cultures on a petri dish? (Typical of great science, the discovery was made when an experiment DIDN’T work).
    Another example would be Lyrica. The molecule was designed as a GABAase inhibitor and Pfizer ONLY lucked out that it had GABA agonist-like effects. Only after the drug was approved did they realize the effects were mostly due to the drugs impact on glutamate pathways.
    These are of course anecdotes, but it does make one wonder how the move towards target-based design has impact drug discovery.

  18. Rick says:

    And let’s not forget the other presumption of the early 90s, combinatorial chemistry, which was supposed to beat the crap out of mere nature for creating chemical diversity. Who needed natural products or natural product chemists, when VCs had declared them dead and decaying and one undergraduate chemisty would soon be able to outdo billions of years of evolution in a single afternoon, while checking his investment portfolio???
    To quote the rock song “Gozdilla”: “history shows again and again how nature points out the folly of men”

  19. drug_hunter says:

    Phenotypic screening is great; I’ve been responsible for starting multiple programs based on phenotypic screens, and they’ve been very successful.
    But structure-based programs can be great too – would we have dozens of drugs for HIV and HCV and HMGCoA and ACE and {fill in the blank} without such approaches? No.
    So it isn’t either/or. Use the best approach for the problem at hand. And if you are following a single target (like ACE) make sure you’re also following phenotype.
    Make sense??

  20. processchemist says:

    The effect of Blue Oyster Cult as soundtrack for a presentation of a phenotypic screening focused biotech should be interesting…

  21. Anonymous BMS Researcher says:

    In Virology, phenotypic screens are as Derek notes particularly straightforward to perform. In addition, once you have found some tool compounds from a phenotypic screen, you can then select for resistance mutations and sequence them to identify the molecular target(s) you are hitting. Once you have made resistance variants, you can drive SAR with a range of assays from binding to phenotype on wild-type and resistant viruses.

  22. newnickname says:

    When the biotech where I worked had programs that were going nowhere (but down) fast, I read Gerald B. Dermer’s “The Immortal Cell; Why cancer research fails”. As I recall, Dermer basically says to get out of the plates and into the whole animal. We are pretty good at killing, growing or otherwise controlling cells in a dish using small molecules but those molecules usually fail to become drugs. The reductionist molecular approach generates a lot of scientific knowledge but the real, whole systems are a heck of lot more complicated.

  23. NoDrugsNoJobs says:

    #19 – Drug Hunter, right on – couldn’t agree more!

  24. CMCguy says:

    I also concur that #19 drug_hunter has the correct mindset of matching approach to problem/data however wonder how many places support this type of empirical based efforts today? That used to be customary research pathway of most Pharmas yet commentary suggests its seems now to be camps of “one or the other” with the past decades dominated by target drivers. At the heart would implicate reflects need to provide a “simple & clear drug discovery” message to Management & Investors who don’t really comprehend or care as long as profits generated.
    #18 Rick combichem, which after all was motivated to feed HTS/targets, was not so much about competing with natural products in terms of diversity but did become dogmatic in attempting to cover wider chemical space with simpler molecules. Fallacy IMO developed with combichem was two-fold , libraries contained too many “reactive compounds” that had false positives or conversely designed to be so “drug-like” decreased the odds of a hit. Natural products largely fell out of favor before the 80s as a viable source of drugs (mostly because “success of SM drugs”) which was too bad. The real issue with both combichem and Natural products is rarely immediately pop out the actual compound that would be good drug and requires solid med chem (tied with pharmacology and biology then process/clinical) to translate.

  25. Frank says:

    The problem with trying to do phenotypic drug discovery in Big Pharma is institutional resistance to it- especially from chemists.
    The majority of scientists leading projects in drug discovery today have spent 20 years doing target-based work so a true phenotypic approach is stepping out of the comfort zone for them. Assay flowscheme, cheminformatics,SAR, tox, all need to be handled differently and there is no cookie-cutter approach- each system has different issues.
    Many people would rather work on a GPCR or enzyme target that they know will probably get them a candiate- even if they know this compound is very unlikely to become a drug.

  26. Rick says:

    Frank (#25), I agree that the institutional resistance factor is very large. However, I wouldn’t lay it primarily on the chemists. The origins of this problem began with biologists who promised that the “-omics” du jour would deliver targets that made drug discovery faster, cheaper, better many times over.
    I also agree with the view that some have expressed that it’s not the target based focus per se that’s the problem. The problem is that it literally swept aside non-target/non-mechanism screening, which were still working just fine at the time. Why? Because there was money to be made on new things!
    But the lion’s share of the blame goes on a business culture that simply does not know what it does now know and believes that what it does not know is unimportant or can be bought with the assistance of scientists – sometimes called “key opinion leaders” – whose opinions and scientific integrity can be bought. Examples of those unscrupulous scientists can be seen in Derek’s more recent posts. But hey, they made money, so it must be all good!!!

  27. TJR says:

    Interesting post and discussion thread.
    We’ve done both target and phenotypic screens with the usual mixture of success and failure for both.
    One question I do have though, that goes beyond the initial discovery phase. It’s based around many people’s view that if you don’t know the target then for many diseases you’ll not get approval for the drug or even for clinical trials in some cases. What I mean is will the regulatory authorities and clinicians be satisfied with “it hits this pathway” or “we see this effect but exactly which enzyme is critical is not certain”?
    Any personal experience of this out there? Also, from what we have seen, hoping to be able to pick the target out at a later stage, thus resolving the question is a strategy with few if any guarantees.

  28. Rick says:

    TJR (#27),
    I am very glad you asked these questions because I have run across much misinformation on this, which is unfortunate because: a, the agency’s position is actually quite simple, clear and easy to find; and b, this myth can result in a lot of misdirected time, energy and priorities. In the therapeutic areas with which I am familiar (incl. infectious diseases, metabolic disease and cancer) the view that “if you don’t know the target then for many diseases you’ll not get approval for the drug or even for clinical trials in some cases.” is NOT correct. This is consistent with FDA guidance documents (available online), which clearly spell out that knowledge of mechanism at some level (such as what you described) is nice to know, but knowing the specific molecular target is not essential for clinical trials or approval. The FDA is unambiguous that their focus is on safety and efficacy, not mechanistic novelty. The only people I have run across who say molecular target is crucial to FDA approval are those who have a buck to make off persuading you that’s the case.
    Your second question, “will the regulatory authorities and clinicians be satisfied with “it hits this pathway” or “we see this effect but exactly which enzyme is critical is not certain”? “, must be answered in two parts. As I said above, regulatory authorities (in my experience and from discussions I’ve had with regulatory experts) is that such information is more than sufficient and that even less information is generally fine, so long as it is clearly and scientifically presented. With respect to clinicians, there is a spectrum because many of them are also researchers and therefore prefer to work at the cutting edge of their speciality, which results in their favoring certain candidates over others based on their target/mechanism.
    I’m not clear about your final point about picking out the target at a later stage. If, by “picking”, you mean selecting the target you will hit and use to cure the disease, the point of Derek’s post is that it’s been less successful than approaches that begin mechanistically agnostic. If, by “picking”, you mean “eventually figuring out how the drug works at some point pre- or post-approval”, the record thus far is overwhelmingly clear: the mechanisms of almost all drugs was resolved at the target level later than the founding drug in the class was in the market. By the way, this includes some drugs where we actually thought we knew the mechanism were chasing, the beta lactams and aspirin being particularly well-studied examples.
    NONE of this is to say that understanding the mechanism is a waste of time and energy. To the contrary, it has been quite helpful at different points in the process, often in ways that we didn’t anticipate. But giving the the primacy it is currently accorded doesn’t seem to be either wise or justified. We’re just not that clever yet.

  29. SP says:

    We need to look at the other side of the coin. We need to compare the failure rates of target screening and phenotype screening. Is there any significant difference between them? Then we will see full picture and see the advantage of either one of them.

  30. HAR says:

    Rick (#28) – Very interesting post Rick. Your comment shade some lights on the regulatory legends that often influence the decision making process in Biotech companies!
    May I ask you to post the references/link to the FDA guidance documents you are referring to, in particluar those relevant for the oncology area?

  31. CG says:

    Rick (#28) and TJR (#27) – thanks for the interesting comments, looking into the long-term effects of a specific research strategy
    Rick, would it be possible to share the hyperlink(s) to the specific Guidance documents you are referring to? In the ocean of FDA Draft Guidances and Guidances I can only find indirect references to what you mention (such as ‘..if the molecular target is known…) which would make one assume that it is acceptable for a molecular target to be NOT known, but nothing positively stating this, which would be really useful.

  32. CMCguy says:

    #28 Rick although I largely agree with what you state, particularly relative to consultant inspired misconceptions, I would suggest there are some who hold more firmly to having defined MOA approach within various Agencies/areas and in the VC/Clinical community. The consequences are that can often get more questions to deal with throughout and have to “do more work” to prove a drug works. In the end that may not be so bad as scientifically while don’t like to say “we are not really sure how it works” (in spite of wonderful PowerPoints that offer modes) however that extra work can (should) provide additional data that shows drug “does work in Clinical meaningful way”. “Good Clinical data” usually still wins for approvals and the more that can be provided the more people will be convinced.

  33. Rick says:

    CMCGuy (#32) I agree with you. The near-mandate to “know” the mechanism (which often looks better in PowerPoint than in the actual lab) comes from places other than the FDA – such as funding agencies, VCs, investors and others – who have a vested (literally) interest in knowing that their speculation (financial or intellectual) on a particular approach worked as advertised. Clinicians sometimes have a preference for knowing the mechanism and favoring one mechanism over another because of their specific research interests.

  34. Rick says:

    Har (#30) and CG (#31) and anyone else who’s interested in links to the FDA’s interest – or lack thereof – in MOA,
    First of all, lest you blame me for the frustration you WILL encounter upon wading into the dense thicket of government regulatory documents, let me warn you that you have to plow through a lot of non-intuitively named links (at least to me) AND basically what you will be looking for is NOT a statement that “mechanism is unimportant”, but rather the LACK of a statement that knowing mechanism is required and specific rules about what constitutes acceptable data. It’s easiest to recognize this lack when you compare it with the (relatively speaking) sharp crystal clarity of requirements for toxicology information.
    To find all the regulatory guidance documents, start at:
    HOWEVER, I suggest you start by looking at the document with the ungainly title “Guidance for Industry Content and Format of Investigational New Drug Applications (INDs) for Phase 1 Studies of Drugs, Including Well-Characterized, Therapeutic, Biotechnology-derived Products” ( In other words “What you need in your IND”.
    Go to the section entitled “Pharmacology and Drug Distribution [21 CFR312.23(a)(8)(i)]”, which states: “This section should contain, if known: 1) a description of the pharmacologic effects and mechanism(s) of actions of the drug in animals, and 2) information on the absorption, distribution, metabolism, and excretions of the drug. The regulations do not further describe the presentation of these data, in contrast to the more detailed description of how to submit toxicologic data. […] If this information is not known, it should simply be so stated. To the extent that such studies may be important to address safety issues, or to assist in evaluation of toxicology data, they may be necessary; however, lack of this potential effectiveness information should not generally be a reason for a Phase 1 IND to be placed on clinical hold.”
    The two salient phrases in these instructions are “IF KNOWN” and “lack of this potential effectiveness information should not generally be a reason for a Phase 1 IND to be placed on clinical hold.” For a official regulatory document, it doesn’t get much clearer than that. If you still have the willpower and time to trudge through other regulatory guidances, found on the first FDA link I provided, you will see similarly relatively non-committal statements regarding mechanism of action, IF MECHANISM OF ACTION/TARGET IS MENTIONED AT ALL, all of which essentially say it’s nice to have (as opposed to must-have) and there’s no stipulation as to how much detail to get into.
    Happy exploring!

  35. Anon says:

    Drugs approved 1999 to 2008 come from Discovery programs 1989 to 1998. Don’t know the percentage of target vs. phenotypic approaches in that time. The 90s were pre-genome, pre-intelligent counter-screens of variants and homologs etc. So, not really relevant to current target-based approaches.
    Not that I am at all against phenotypic screens! I just think it’s meaningless to say one approach is always better than the other. Do the screen that makes sense.

  36. Smbswh says:

    Anon (35), if you read the original article you’ll see that the majority of drugs approved between 1999 and 2008 were actually from target-based programmes. However, most of these were follower-drugs where the target was well established and validated clinically. For first-in-class drugs there was a clear bias toward phenotyopic approaches. This was particular true in areas such as CNS (1 first-in-class drug from a target-based approach).
    This suggests it has nothing to do with pre- vs post-genome era but simply to being able to identify good targets. A good target would be defined as one that has been established with clinical studies, using a drug or drug(s) that originated with a phenotypic approach.
    In the CNS area this all also strongly implies that our knowledge of disease mechanisms in the brain is extremely limited.

Comments are closed.