Skip to Content

Three Not-So-Reproducible Papers

You may well remember Amgen’s statement in 2012 about how many academic papers they were having trouble reproducing. Not everyone has taken it seriously, since they didn’t provide specific details, just an overall count. (On the other hand, a lot of people inside the drug industry just nodded their heads, having had similar – and similarly nonpublishable – experiences themselves).

Now the company is back with some chapter and verse, although I wish they had even more. This newly launched site at Faculty of 1000 is going to be dedicated to this sort of thing, and it will be very interesting to see how it fares. The first entry is on GPR21 knockout mice. These were reported in 2011 and 2012 to show increased insulin sensitivity and glucose tolerance – and that first reference, by the way, is from a team at Amgen itself (the second is from UCSD and Scripps). It turns out that the earlier knockout animals also had another gene disabled (Rabgap1), and generating the knockout through another technique (TALEN) gave a cleaner animal model that unfortunately shows no such effects on glucose and insulin.

The second entry refers to the ubiquitin-specific protease USP14, which was reported in 2010 as a possible target in neurodegenerative disease. Inhibiting the protease with a small molecule, it was believed, would speed up the degradation of harmful proteins like tau (which otherwise would be de-ubiquitinated and saved from destruction). The Amgen group, though, could not confirm any role for USP14 in degradation of such proteins, despite several attempts. This adds to a 2012 report that USP14-deficient mice didn’t seem to show any effects on tau levels, either.

And finally, there’s a follow-up on the effects of an LXR/RXR agonist on the levels of beta-amyloid in the brain and CSF. (My own opinion: that’s just what you need to increase the success rate of an Alzheimer’s program – go after a centrally acting nuclear receptor target. Not only are you stumbling around in a blacked-out basement, you’ve tied your feet together as well). This was the 2012 bexarotene paper, which I blogged about here. That one picked up criticism from several other labs who could not reproduce its findings, so this Amgen report (they couldn’t, either) doesn’t add much to the story, from what I can see.

So while I’m glad to see these, and also very glad to see the F1000 site that gives these a home, these don’t seem to have as big an impact as one could have expected. Both the bexarotene work and the USP14 results had been dented pretty badly already, and those reports didn’t need this platform to appear in, either. What I’d like to hear about – and I’ll bet I’m not alone – are some of the things that we didn’t know were problematic. Will those start to appear?

16 comments on “Three Not-So-Reproducible Papers”

  1. anoano says:

    Yes, nice start, but far away from the 50 papers Amgen claimed.

  2. Bob Hanky says:

    As far as I know, none of these pathways have human genetic validation. Isn’t that now Amgen’s focus? Also, what would Amgen have done if the results had been positive, try to drug these pathways? What a waste of precious research animals and funds.

    1. Hap says:

      No, the waste was the original research, which probably wasted lots of resources from both the original authors and then from those unfortunate enough to try to use the results.

      If the results of simpler systems don’t work as advertised, then what would you guess the reliability of research in more complex systems would be? I think most people assume it to be better than newspaper reliability, so the idea that the reliability of some published research might be more akin to the claims of supplementeers seems kind of important.

      1. Bob Hanky says:

        I’m not sure the Amgen investor would agree the “waste was the original research”. In the current business environment, at heavyweight companies like Amgen, is there room for “blue sky” research or answering piddling questions of academic reproducibility that appear out of strategic scope (e.g. Amgen’s push for human genetic validation)? I guess that’s one for Derek to answer.

        1. Hap says:

          It depends what they are trying to do. Publishing is probably not looking for a short-term outcome (else they’d validate what they could and use the advantage of what they know doesn’t work to spend money elsewhere) – they are probably trying to get potential partners not to do work in systems that can’t be validated, and perhaps to tell funders that some of the work they are paying for doesn’t actually work (and thus doesn’t achieve the ends they’re paying for). In the long run, that might mean less crap to wade through, and easier path to finding things that can make them money, but it is not an immediate effect. They may also be looking for credit and positive PR from the scientific community, which doesn’t make money now but might help them later.

          1. Bob Hanky says:

            Hap, Amgen need no longer worry about “wading through academic crap”, they bought a human genetics company, deCODE, for a billion bucks.

  3. luysii says:

    These papers have real impact on caregiver’s lives. The very intelligent widow of a good friend now dead of a slowly dementing illness had her hopes raised by the bexarotene paper. Even back then (2014) I had to tell her that people were having trouble reproducing it.

  4. Chris Wood says:

    Nice to see that there is a place to report this kind of reproduction work. I really wish someone would take up the mantle for doing something similar in the biofuels / bioproducts area, where I feel we have allot of the same problems. Good luck to Faculty of 1000, and hopefully this will help clean up the research world a bit.

  5. LiqC says:

    Well the first case seems to be more of a discovery that whatever animal knockout route they took ended up messing up another pathway, which may or may not happen with a pharmacological approach as well.

  6. Anon says:

    @LiqC, it also demonstrates that Amgen itself cannot reproduce its own research sometimes. Which is great to see reported but should be kept in mind when all of the discussions about reproducibility here turn into academic witch hunts and discussions of fraud…science is hard work.

    1. Bob Hanky says:

      Science IS “hard work”, but big pharma science must be focused on drug discovery. For example, on the Neuroscience front, an investor might argue why do these studies at all (the targets of these pathways have poor rationale and tractability)? At the extreme, why even maintain a group of AD researchers when Amgen has already made its best bet on AD with the NVS secretase inhibitor? Folks, come on, if human genetics show amyloid is the bad guy, and if secretase inhibition doesn’t work clinically, what’s the likelihood that USP14 inhibition or LXR/RXR agonism (or anything else) will? Sadly, it sounds like “busy work” to me.

  7. Pay no attention says:

    How do you know they aren’t just trying to get others to ignore something they actually found to be of value?

  8. Kaleberg says:

    Research is an unreliable narrator. I get the impression that most actual researchers discount everything they read to a certain level much as a reader discounts the first person narration in a noir thriller. This means that a lot of research is wasteful, but it seems to be unavoidable. It’s good to see someone tackling this head on.

    I always remember Lewis Thomas’s failure to duplicate the penicillin makes rabbits’ ears floppy result, so this is far from new. (The floppy ears turned to be a seasonal thing.)

    1. Bob Hanky says:

      Kaleberg, “academic” research is an unreliable narrator (publish or perish), but “industrial” research must be much less so. If you pay your gardener to cut the grass, but instead he paints the garage, that’s a misappropriation of funds, no matter how nice the garage now looks. If you invest in a company to discover new medicines, but instead they do a lot of (predictably) dead-end research (at least in the neurology area), that’s a misappropriation of funds, no matter how nice the data looks.

  9. MoMo says:

    I’m with you Bob Hanky. A pharma company using Big Stick Ideology to police the biology journals seems like a Herculean task and not possible by mere mortal men.

    If an employee came to me and wanted to do this I’d fire him on the spot.

    “Make drugs- not data”

  10. steve says:

    Most academic work is done by graduate students and post-docs; i.e., scientists that are not fully trained. Even in top-notch labs assays aren’t validated the way they would be in industry so the fact that a lot of stuff gets published that isn’t reproducible isn’t a great surprise. The fact that industry can also have results that aren’t correct is less likely but obviously can occur when only one assay is used. Science progresses by having multiple labs repeating data using different approaches so they become self-reinforcing.

Comments are closed.