Here are some slides from Anthony Nicholls of OpenEye, from his recent presentation here in Cambridge on his problems with molecular dynamics calcuations. Here’s his cri du coeur (note: fixed a French typo from the original post there):
. . .as a technique MD has many attractive attributes that have nothing to do with its actual predictive capabilities (it makes great movies, it’s “Physics”, calculations take a long time, it takes skill to do right, “important” people develop it, etc). As I repeatedly mentioned in the talk, I would love MD to be a reliable tool – many of the things modelers try to do would become much easier. I just see little objective, scientific evidence for this as yet. In particular, it bothers me that MD is not held to the same standards of proof that many simpler, empirical approaches are – and this can’t be good for the field or MD.
I suspect he’d agree with the general principle that while most things that are worthwhile are hard, not everything that’s hard is worthwhile. His slides are definitely fun to read, and worthwhile even if you don’t give a hoot about molecular dynamics. The errors he’s warning about apply to all fields of science. For example,he starts off with the definition of cognitive dissonance from Wikipedia, and proposes that a lot of the behavior you see in the molecular dynamics field fits the definitions of how people deal with this. He also maintains that the field seems to spend too much of its time justifying data retrospectively, and that this isn’t a good sign.
I especially enjoyed his section on the “Tanimoto of Truth”. That’s comparing reality to experimental results. You have the cases where there should have been a result and the experiment showed it, and there shouldn’t have been one, and the experiment reproduced that, too : great! But there are many more cases where only that first part applies, or gets published (heads I win, tails just didn’t happen). And you have the inverse of that, where there was nothing, in reality, but your experiment told you that there was something. These false positives get stuck in the drawer, and no one hears about them at all. The next case, the false negatives, often end up in the “parameterize until publishable” category (as Nicholls puts it), or they get buried as well. The last category (should have been negative, experiment says they’re negative) are considered so routine and boring that no one talks about them at all, although logically they’re quite important.
All this can impart a heavy, heavy publication bias: you only hear about the stuff that worked, even if some of the examples you hear about really didn’t. And unless you do a lot of runs yourself, you don’t usually have a chance to see how robust the system really is, because the data you’d need aren’t available. The organic synthesis equivalent is when you read one of those papers that do, in fact, work on the compounds in Table 1, but hardly any others. And you have to play close attention to Table 1 to realize that you know, there aren’t any basic amines on that list (or esters, or amides, or what have you), are there?
The rest of the slides get into the details of molecular dynamic simulations, but he has some interesting comments on the paper I blogged about here, on modeling of allosteric muscarinic ligands. Nicholls says that “There are things to admire about this paper- chiefly that a prospective test seems to have been done, although not by the Shaw group.” That caught my eye as well; it’s quite unusual to see that, although it shouldn’t be. But he goes on to say that “. . .if you are a little more skeptical it is easy to ask what has really been done here. In their (vast) supplementary material they admit that GLIDE docking results agree with mutagenesis as well (only, “not quite as well’, whatever that means- no quantification, of course). There’s no sense, with this data, of whether there are mutagenesis results NOT concordant with the simulations.” And that gets back to his Tanimoto of Truth argument, which is a valid one.
He also points out that the predictions ended up being used to make one compound, which is not a very robust standard of proof. The reason, says Nicholls, is that molecular dynamics papers are held to a lower standard, and that’s doing the field no good.