Skip to Content

Analytical Chemistry

An HIV Structure Breakthrough? Or “Complete Rubbish”?

Structural biology needs no introduction for people doing drug discovery. This wasn’t always so. Drugs were discovered back in the days when people used to argue about whether those “receptor” thingies were real objects (as opposed to useful conceptual shorthand), and before anyone had any idea of what an enzyme’s active site might look like. And even today, there are targets, and whole classes of targets, for which we can’t get enough structural information to help us out much.
But when you can get it, structure can be a wonderful thing. X-ray crystallography of proteins, and protein-ligand complexes has revealed so much useful information that it’s hard to know where to start. It’s not the magic wand – you can’t look at an empty binding site and just design something right at your desk that’ll be a potent ligand right off the bat. And you can’t look at a series of ligand-bound structures and say which one is the most potent, not in most situations, anyway. But you still learn things from X-ray structures that you could never have known otherwise.
It’s not the only game in town, either. NMR structures are very useful, although the X-ray ones can be easier to get, especially in these days of automated synchroton beamlines and powerful number-crunching. But what if your protein doesn’t crystallize? And what if there are things happening in solution that you’d never pick up on from the crystallized form? You’re not going to watch your protein rearrange into a new ligand-bound conformation with X-ray crystallography, that’s for sure. No, even though NMR structures can be a pain to get, and have to be carefully interpreted, they’ll also show you things you’d never had seen.
And there are more exotic methods. Earlier this summer, there was a startling report of a structure of the HIV surface proteins gp120 and gp41 obtained through cryogenic electron microscopy. This is a very important and very challenging field to work in. What you’ve got there is a membrane-bound protein-protein interaction, which is just the sort of thing that the other major structure-determination techniques can’t handle well. At the same time, though, the number of important proteins involved in this sort of thing is almost beyond listing. Cryo-EM, since it observes the native proteins in their natural environment, without tags or stains, has a lot of potential, but it’s been extremely hard to get the sort of resolution with it that’s needed on such targets.
Joseph Sodroski‘s group at Harvard, longtime workers in this area, published their 6-angstrom-resolution structure of the protein complex in PNAS. But according to this new article in Science, the work has been an absolute lightning rod ever since it appeared. Many other structural biologists think that the paper is so flawed that it never should have seen print. No, I’m not exaggerating:

Several respected HIV/AIDS researchers are wowed by the work. But others—structural biologists in particular—assert that the paper is too good to be true and is more likely fantasy than fantastic. “That paper is complete rubbish,” charges Richard Henderson, an electron microscopy pioneer at the MRC Laboratory of Molecular Biology in Cambridge, U.K. “It has no redeeming features whatsoever.”
. . .Most of the structural biologists and HIV/AIDS researchers Science spoke with, including several reviewers, did not want to speak on the record because of their close relations with Sodroski or fear that they’d be seen as competitors griping—and some indeed are competitors. Two main criticisms emerged. Structural biologists are convinced that Sodroski’s group, for technical reasons, could not have obtained a 6-Å resolution structure with the type of microscope they used. The second concern is even more disturbing: They solved the structure of a phantom molecule, not the trimer.

Cryo-EM is an art form. You have to freeze your samples in an aqueous system, but without making ice. The crystals of normal ice formation will do unsightly things to biological samples, on both the macro and micro levels, so you have to form “vitreous ice“, a glassy amorphous form of frozen water, which is odd enough that until the 1980s many people considered it impossible. Once you’ve got your protein particles in this matrix, though, you can’t just blast away at full power with your electron beam, because that will also tear things up. You have to take a huge number of runs at lower power, and analyze them through statistical techniques. The Sodolski HIV structure, for example, is the product of 670,000 single-particle images.
But its critics say that it’s also the product of wishful thinking.:

The essential problem, they contend, is that Sodroski and Mao “aligned” their trimers to lower-resolution images published before, aiming to refine what was known. This is a popular cryo-EM technique but requires convincing evidence that the particles are there in the first place and rigorous tests to ensure that any improvements are real and not the result of simply finding a spurious agreement with random noise. “They should have done lots of controls that they didn’t do,” (Sriram) Subramaniam asserts. In an oft-cited experiment that aligns 1000 computer-generated images of white noise to a picture of Albert Einstein sticking out his tongue, the resulting image still clearly shows the famous physicist. “You get a beautiful picture of Albert Einstein out of nothing,” Henderson says. “That’s exactly what Sodroski and Mao have done. They’ve taken a previously published structure and put atoms in and gone down into a hole.” Sodroski and Mao declined to address specific criticisms about their studies.

Well, they decline to answer them in response to a news item in Science. They’ve indicated a willingness to take on all comers in the peer-reviewed literature, but otherwise, in print, they’re doing the we-stand-by-our-results-no-comment thing. Sodroski himself, with his level of experience in the field, seems ready to defend this paper vigorously, but there seem to be plenty of others willing to attack. We’ll have to see how this plays out in the coming months – I’ll update as things develop.

34 comments on “An HIV Structure Breakthrough? Or “Complete Rubbish”?”

  1. Anonymous says:

    Even X-ray crystallography is subject to over-fitting data, therefore one has to keep a portion of data out of the fitting algorithms to assess the real level of fit. Similarly, this group should have used only a partial model, and then look what happens outside the model to see if there is anything more than white noise.

  2. MDACC Grad says:

    This is one of those situations where it would be great if the investigator could invite others out to their lab and jointly workout what they did and how it was accomplished (…assuming the PI is familiar with how his postdocs/grad students accomplished this). But grant funding and competition are going to restrict this to political-like arguments. I really doubt anyone will be discussing hard science (specific methods, calculations, buffers, etc.) in these correspondence.

  3. ScientistSailor says:

    I love a good cat fight first thing in the morning…

  4. luysii says:

    I’ve always avoided CryoEM papers. The resemble Rohrschack tests more than anything else.

  5. PPedros says:

    The sentence “Just make up a Cryo-EM” comes to mind… 😀

  6. MoMo says:

    Wasnt this funded by the NIH? Our tax dollars?
    Its time for an Inquistion!

  7. jbosch says:

    CryoEM is very powerful if done right and very helpful.
    I’m linking here to my former PhD advisor:
    Think of the ribosome structures for example before we had the nice high resolution Xray structures.
    But you can average noise and generate your template what you used for a search. Usually you take your 1000000 particles and generate sub classes as you end up with random orientations of your protein in the vitreous ice. And then you should have a subset of images which you have not touched and verify that the subclasses identified on the first set are re-identified on the second set.

  8. Casual Observer says:

    “In an oft-cited experiment that aligns 1000 computer-generated images of white noise to a picture of Albert Einstein sticking out his tongue, the resulting image still clearly shows the famous physicist.”
    Does anyone have a pointer to that experiment? Would really love to see it

  9. Nekekami says:

    Oooh, oooh
    *sells popcorn and umbrellas for the upcoming scientist feces slinging match*

  10. Toad says:

    As a related aside, I’d like to reiterate to the community the point that #1 Anonymous brings up: the current state of X-ray crystallography quality.
    During structure-based med chem design work, we have found a high percentage of errors and mis-assignments in the Protein Data Bank. For example, one of the most common is the reversal of the O and N atoms in the carboxamide sidechain functionality in asparagine and glutamine residues. Typically, the correct assignment can be made based simply visually based on the H-bonding network commonly seen around these groups.
    If you are using a pdb structure as an input for anything, please make sure you carefully analyze it yourself prior to use, visually and using standard computational tools. A lot of junk is just getting thrown into the database these days.

  11. Anonymous says:

    @11: Indeed, C, N and O atoms are difficult to distinguish by X-ray crystallography, because they are similar in electron density, while H atoms never show up. So assigning these atoms must be done carefully (and checked) when there are more than two orientations in pseudo-symmetric amino acid side chains, like His, Asn and Gln, by looking at their interactions with neighbouring atoms.
    However, my main point was that over-fitting of data is the biggest problem, and can only be avoided by looking at the correlation between observed results and a portion of data that has not been used in refinement. It seems this Cryo-EM group has overlooked this issue – a schoolboy error!

  12. Fitting data to cryo-EM seems to be a good example of what psychologist and skeptic Michael Shermer calls “patternicity”, the tendency engineered into our primitive, Savannah-dwelling brains to ascribe signal to noise to avoid making false negative errors that could have weeded us out of the gene pool. It’s like seeing the Virgin Mary in a piece of wood bark.

  13. Imaging guy says:

    # 9.Casual Observer
    This article shows how that is done.
    “A method for the alignment of heterogeneous macromolecules from electron microscopy”, Maxim Shatsky et al, Journal of Structural Biology 166 (2009) 67–78.

  14. Anonymous says:

    @13: That’s one of my big concerns with Big Data, and we see it all the time in post-hoc clinical data analysis, leading to hopes of success in Phase 3, based on patterns in Phase 2 data that is nothing more than noise.

  15. anon says:

    a little bit dubious to say that protein crystallography is easier. crystallization is a pretty big hurdle.

  16. Anonymous says:

    @17: Back in my PhD days they used to call me Buffer Boy, because I had to make up over 3,000 buffers to get my first decent crystals. 🙂

  17. Derek Lowe says:

    #9, I’ve put up a link to the Einstein-picture result. It’s in figure 2 of the paper (open access).
    #1, 11, 12: absolutely right. X-ray crystals are to be given the fishy eye as well, thus that “not the magic wand” link in the second paragraph. I think that’s why cryo-EM is such a beast. Think about how many things can go wrong with, say, a 2.4 A x-ray structure. And a good cryo-EM structure is what, 9 to 12 angstroms? Everyone’s going wild about this possible 6 A resolution, and for someone that’s used to X-ray structures, that’s like groping around in the dark with ski gloves on. I feel for everyone doing that work; they’re right on the edge of what can be accomplished.

  18. Casual Observer says:

    @14: Thanks very much, it’s a terrific demonstration of how scarily powerful overfitting your data can be. (Applies to many other things as well as scientific data, I think.)

  19. Casual Observer says:

    And @18, thanks Derek as well!

  20. NMH says:

    @17: well, at least you got decent crystals. I never did, except one, that had a unit cell for a salt. And I have the publication record to show for it. That’s why I got out of crystallography.
    Beware of any PI who has a specific aim to purify a protein and get crystals, and wants to convince a grad student to do it.

  21. Anonymous says:

    @21: It helped to be a masochist. 🙂
    At least I had the computing power and algorithms I needed to solve the sructure. My PI was Max Perutz, who had to use punch cards to store all the X-ray data (and a small army of attractive female assistants to manage them all).

  22. Anonymous says:

    @22: I was a masochist. I purified my protein in a glove box in a cold room, in the middle of the night.
    I have not been the same ever since. 🙁

  23. hn says:

    Crystallization is not always a big hurdle. Some proteins are easy to crystallize. Yours may just be one of the lucky ones. Don’t give up before you try.

  24. Sili says:

    Michael Shermer calls “patternicity”

    Why am I not surprised he felt the need to make up his own word for “pareidolia”.

  25. Secondaire says:

    Speaking of dodgy crystal structures, there is a sub-group within my group who is collaborating to try and get a structure of an inactivated enzyme + inactivator complex , and the collaborators (who seem to know not much of anything about enzymes) keep doing things like trying to obtain structures without the cofactor present (d’oh!), or under conditions that are shown to NOT actually inactivate the enzyme, and then they keep wondering why what comes out the other end makes no sense.

  26. cliffintokyo says:

    If Cryo-EM provides some structural information about HIV surface proteins gp120 and gp41 that we do not/cannot get using another protein structure elucidating technique, then we need to have a very transparent and serious discussion about the validity of the work by the Sodroski group, and perhaps the validity of Cryo-EM technology –
    because we want to use such important structural information for med chem design purposes.
    If not, we do not need to get over-excited about the degree of resolution!
    @#27: This seems to more about over-interpreting data, judging from the comments, although I am not familiar with this technology.

  27. Anonymous says:

    @28: The article also explains why and how we tend to over-interpret data.
    I posted it only because it seems relevant to this discussion…

  28. zmil says:

    Sat in on a presentation of this work a little over a year ago. I have never, ever seen such utter disbelief of a presentation. They brought in a non-retrovirologist cryo-EM expert because they weren’t sure anyone else had enough structural background to really critique the project, and he just kept saying, over and over, “you don’t understand, this is *impossible!*” It was interesting at first but then it got so uncomfortable I was wishing it would stop.
    Also, the structure he presented at that meeting? Wayyy more detailed than in this paper. He was pointing out individual residues and crap. Never seen anything like it -well I have, but only with X-ray crystallography. Wish I knew enough to make an informed judgement on the merits, but sadly all I can do is sit in the cheap seats munching popcorn…

  29. AGMMGA says:

    @11: side chain issues and general geometry issues for the protein can be sorted out pretty easily by algorithms nowadays. If you don’t wanna go through th pain of checking everything yourself, check out pdb_redo ( They re-refine old data with the latest algorithms, which spot these problems and automatically correct them. Also, molprobity ( might help to flag any problematic areas.

  30. Anonymous says:

    Maybe we could get a 1-Angstrom molecular structure if we just merge a trillion brass rubbings?

  31. Spoons says:

    This reminds me of my Biochemistry class. Since biochemistry was a sub major of chemistry our professor went to great lengths to present papers where some biologists tried to do a lot of hand waving and yadda-yadda when it came to the important underlying chemistry elements of a biology paper. Now I realize she was trying to get our BS-detectors trained up. Same thing with my instrumental analysis class.

  32. JTA says:

    Pop corn time is back!
    In this week’s PNAS, three lengthy critiques are published (by Richard Henderson, Marin van Heel, and Sriram Subramaniam), as well as a lengthy reply from the authors. From my quick reading, the original authors confront the critiques only selectively. For example Henderson’s Figure 2I-L are damning enough on their own but are not addressed in the author’s reply.

Comments are closed.