So, are half the interesting new results in the medical/biology/med-chem literature impossible to reproduce? I linked earlier this year to an informal estimate from venture capitalist Bruce Booth, who said that this was his (and others’) experience in the business. Now comes a new study from Bayer Pharmaceuticals that helps put some backing behind those numbers.
To mitigate some of the risks of such investments ultimately being wasted, most pharmaceutical companies run in-house target validation programmes. However, validation projects that were started in our company based on exciting published data have often resulted in disillusionment when key data could not be reproduced. Talking to scientists, both in academia and in industry, there seems to be a general impression that many results that are published are hard to reproduce. However, there is an imbalance between this apparently widespread impression and its public recognition. . .
Yes, indeed. The authors looked back at the last four years worth of oncology, women’s health, and cardiovascular target validation efforts inside Bayer (this would put it right after they combined with Schering AG of Berlin). They surveyed all the scientists involved in early drug discovery in those areas, and had them tally up the literature results they’d acted on and whether they’d panned out or not. I should note that this is the perfect place to generate such numbers, since the industry scientists are not in it for publication glory, grant applications, or tenure reviews: they’re interested in finding drug targets that look like they can be prosecuted, in order to find drugs that could make them money. You may or may not find those to be pure or admirable motives (I have no problem at all with them, personally!), but I think we can all agree that they’re direct and understandable ones. And they may be a bit orthogonal to the motives that led to the initial publications. . .so, are they? The results:
“We received input from 23 scientists (heads of laboratories) and collected data from 67 projects, most of them (47) from the field of oncology. This analysis revealed that only in ~20–25% of the projects were the relevant published data completely in line with our in-house findings. In almost two-thirds of the projects, there were inconsistencies between published data and in-house data that either considerably prolonged the duration of the target validation process or, in most cases, resulted in termination of the projects. . .”
So Booth’s estimate may actually have been too generous. How does this gap get so wide? The authors suggest a number of plausible reasons: small sample sizes in the original papers, leading to statistical problems, for one. The pressure to publish in academia has to be a huge part of the problem – you get something good, something hot, and you write that stuff up for the best journal you can get it into – right? And it’s really only the positive results that you hear about in the literature in general, which can extend so far as (consciously or unconsciously) publishing just on the parts that worked. Or looked like they worked.
But the Bayer team is not alleging fraud – just irreproducibility. And it seems clear that irreproducibility is a bigger problem than a lot of people realize. But that’s the way that science works, or is supposed to. When you see some neat new result, your first thought should be “I wonder if that’s true?” You may have no particular reason to doubt it, but in an area with as many potential problems as discovery of new drug targets, you don’t need any particular reasons. Not all this stuff is real. You have to make every new idea perform the same tricks in front of your own audience, on your own stage under bright lights, before you get too excited.