Skip to main content

Don’t Let Humans Pick the Experimental Conditions?

When chemists have a wide range of reactants to choose from to make new compounds, how do they choose which ones to use? “Not randomly” is the answer, even when perhaps it should be. This effect has been noted in medicinal chemistry, where the choice of building blocks (not to mention reactions) for analog synthesis is influenced both by what people tend to usually use and by the related issues of what’s easily available from suppliers or what’s already on the shelf. This new paper (from researchers at Haverford and Fordham) documents the same sorts of effects in inorganic chemistry, specifically the synthesis of metal oxide compounds with amines.

Since the successful outcome of these experiments is an X-ray crystal structure, the field can be studied pretty well by looking at what’s been deposited in the Cambridge Structural Database. And the distribution of amines therein follows a rough 80/20 Pareto distribution:

The CSD includes the structures of 5,010 amine-templated metal oxides that contain 415 unique amines. The top 17% commonly reported amines (70 individual molecules, the ‘popular’ amines) are found in 79% of the structures (3,947 distinct CSD entries), and correspondingly, the remaining 83% (345 molecules, the ‘unpopular’ amines) of amines are found in just 21% of the structures (1,063 entries; see Fig. 1)

The amine statistics are completely consistent with human bias in the selection process, as seen in chemistry and in many other fields. There’s a gap, a lag between person-to-person communication or personal experience and the incorporation of that sort of information into the broadly used knowledge of a process, and that gap is one of the things that seems to produce 80/20 distributions. Other explanations are easier than usual to rule out in this particular example – as the authors note, there are (1) no particular technical reasons (compatibility with the apparatus, etc.) why one would choose certain amines, (2) no rules about desirable final products that would favor choosing some over others, and (3) the distribution of  amines doesn’t seem to follow any particular cost or commercial availability relationship. A set of 55 structurally varied amines that were all of basically equal cost and ease of ordering had 27 on the popular list, 16 on the unpopular list, and 12 that were so unpopular as to be absent from CSD records entirely. These categories of reasonable bias are referred to in the literature on the subject of choice as “efficient causes”, “final causes”,  and “material causes”.

There’s one category left, “formal causes”, which in this case would be some property of the popular amines that just make them more likely to generate useful crystals (that is, they look more successful because they really are more successful). The paper rules this one out by experiment: the authors took that set of 55 amines mentioned and ran 10 reactions on each, with experimental conditions set to vary randomly within ranges known in the broader literature. What they found was that any single reaction had the same chances of success (an X-ray quality crystal) regardless of the popularity of the amine used – so it’s not that there are winning go-to amines that you can reach for (although people apparently think that there are!)

The paper goes on to illustrate the implications for machine-learning models. This situation gives a software model a real chance to outperform the human chemists, so long as the humans setting up the model don’t import their biases into it at the beginning. The paper shows that models trained on human-selected data for this metal-oxide-amine crystallization process underperform in every way compared to less-biased ones. Interestingly, in the 110 reactions predicted by both models, the two disagree in 23 cases, and in every one of them the model based on human selections predicts failure, while the random model predicts success. This may well be experimental proof of the human tendency towards “Nah, that’s not gonna work” style thinking (loss-aversion bias, in other words, as famously studied by Tversky and Kahneman). In case you’re wondering, in this example there were 16 successes correctly called by the random model, and 7 failures correctly called by the human-biased one. Larger runs indicate that the random models are better at calling successes in the less-explored regions of reaction space as well, and are thus better at suggesting new experiments to produce unknown materials.

So we don’t know as much as we think we do, in such cases. For mental exercise, contrast this situation with the “wisdom of crowds” idea, and then move up one level of abstraction to try to decide what heuristic you’d use to choose (in any given situation) between trusting to that or setting up a system that tries to remove any traces of what the crowd thinks is a good idea. When has the crowd (or the smeared-across-time crowd of human experience) discovered useful rules about those efficient causes, formal causes and the rest of them, and when has it just been taking shortcuts that have no basis in reality? This system has the advantage of being subject to relatively straightforward experimental proof; you can show that the anthropogenic biases are actually slowing down discovery. How about the cases where such experimentation is difficult or even impossible?

8 comments on “Don’t Let Humans Pick the Experimental Conditions?”

  1. BP says:

    It is probably something we all suspected at some point. It is nice to see someone actually sat down with data and analyzed the human bias. What is interesting from Kahneman and Tversky’s work is even experts are not free from the bias. Scientists who, one could argue, should know better can be swayed by biases innate to human decision-making. The good news is the bias can be diminished greatly (maybe even eliminated but I don’t remember) through simple mental exercises.
    We all learn how to design experiments, and we can do it well. The problem is, it always will be, life is not simple. You have some bosses who always want more compounds, and you have two kids in private school. It’s easier to do 5 suzuki couplings to collect a paycheck than scout a new scaffold.
    By the way, I don’t think the human tendency towards “Nah, that’s not gonna work” style thinking is exactly the loss-aversion bias. The loss aversion bias is more like “I hate losing more than I ever wanna win” as Brad Pitt said in moneyball.

  2. tlp says:

    Each individual researcher doesn’t really care on the daily basis about the field overall, but cares about their career perspective or immediate project success (e.g. graduation). Which is exact opposite of entrepreneurs’ optimism.

    The other factor may be that humans are really bad at evaluating influence of more than one variable at a time.

  3. KazooChemist says:

    Paywall, so I can’t (won’t) check for myself. Did the authors ask any of the chemists why they chose the amines that they used? If I was interested in studying the properties of a series of metal oxide amine salts and I was focusing on the metal I think I would try to hold the amine constant. If I wanted to compare a compound made in my lab to another in the literature it would be natural to use the same amine. There are numerous reasons for choosing a particular amine beyond price and availability.

  4. RM says:

    I’m none too surprised at the machine learning results. The problem here is that with the CSD you don’t actually have true “negative” results – you don’t actually know that the amine “failed” at crystallization, you only know that it hasn’t been annotated as succeeding.

    That sort of setup is hell on conventional machine learning techniques which assume you have accurate labels for the positive and negative sets you’re using to train your classifier, and that – equally important – the positive and negative sets are being drawn from the same distribution. (That is, whether you see any particular instance is independent of it being positive or negative. There’s no “if this would have been negative, I wouldn’t have seen it because the source I’m getting the negatives from implicitly ignored cyclic amines.”)

    See for a recent discussion of the issue regarding protein-small molecule interaction. If you don’t go into things fully aware of the fact that you’re drawing your positives and negatives from different distributions, you’re going to have a bad time. Even if you are aware and attempt to “unbias” your sampling after the fact, you’re likely going to have issues, as unbiasing techniques are anything but simple.

  5. Chris Phoenix says:

    Re useful heuristics vs. “nah not gonna work” and paraphrasing a well-known quote:

    When a distinguished elderly scientist says something is possible, they’re usually right. When they say something is impossible they’re quite often wrong.

  6. a. nonymaus says:

    There’s commercially available, and then there’s “It’s Friday and I want to let this run over the weekend, what’s on the shelf?”

  7. metaphysician says:

    I can’t help but be reminded of the cat whose stock picks outperformed those of a group of experts(?). . .

  8. kjk says:

    I do imagine implicit bias reduction also bieng useful in the workplace.

    VR interview with body language capture onto characters. Everyone is a Wookiee! No worry about those pesky biases!

Comments are closed.