Here are the results of a Nature survey on reproducibility in the scientific literature. They themselves admit that it’s a “confusing snapshot”, but it shows that we’re still arguing about what “reproducibility” means. 52% of the responders (over 1500 scientists) said that there was “a significant crisis”, though, so this issue is on people’s minds.
Interestingly, chemists were among the most confidant in the literature of their own field (physics and engineering as well). The medical literature was considered by its own practicioners to be the least reliable, and I think that order is probably about right. (I wrote in a recent column for Chemistry World about how chemistry might not have it as bad in this regard, <s>but that one’s not online yet</s>). At the same time, chemists had the highest proportion of respondents who said that they’d been unable to reproduce someone else’s experiment.
I don’t think that’s necessarily a contradiction, though. Chemistry is a field with lower barriers to replication than many others, and we also probably do more replications in general. Biology goes out of date more quickly, but chemists think nothing of trying a thirty or forty-year-old reference if it looks like a useful reaction. And one of the other points the article makes is that “failure to reproduce the paper” can range from something that’s a tissue of lies from beginning to end, all the way to “that reaction doesn’t go in as high a yield as they said it does”. Chemistry has plenty of that, for reasons both forgivable and not. I suspect that many of the non-replications that chemists reported were in the venial category, which still leaves room to believe that the literature of the subject itself is largely reproducible.
But sorting discoveries from false leads can be discomfiting. Although the vast majority of researchers in our survey had failed to reproduce an experiment, less than 20% of respondents said that they had ever been contacted by another researcher unable to reproduce their work. Our results are strikingly similar to another online survey of nearly 900 members of the American Society for Cell Biology (see go.nature.com/kbzs2b). That may be because such conversations are difficult. If experimenters reach out to the original researchers for help, they risk appearing incompetent or accusatory, or revealing too much about their own projects.
I’ve had that exact experience – we had problems with some published work, but we couldn’t say anything about it to the original authors without giving too much away. In the times I have contacted authors, though, I’m about 50/50. And the responses are bimodal indeed – the only things I’ve ever had are helpful responses with suggestions about what might be going on, and complete lack of response whatsoever. No one’s ever bothered to get defensive – they just don’t reply. Here’s another problem:
A minority of respondents reported ever having tried to publish a replication study. When work does not reproduce, researchers often assume there is a perfectly valid (and probably boring) reason. What’s more, incentives to publish positive replications are low and journals can be reluctant to publish negative findings. In fact, several respondents who had published a failed replication said that editors and reviewers demanded that they play down comparisons with the original study.
That last part is interesting, and a bit unexpected. Any readers have any similar experiences? I guess the famous “Comment On a Paper by Samir Chatterjee” would have a harder time getting published today (if you don’t have access, just the abstract will give you the flavor of the thing). I suspect that John Cornforth bypassed the standard review process for that one!
When asked why we have such problems, the Nature respondents listed a number of very likely causes: selective reporting of results, pressure to publish, insufficient replication inside the lab before publication, lack of statistical power in the first place, and so on. Outright fraud was about halfway down the list. Unfortunately, when people outside of science hear about a reproducibility crisis, that’s what they tend to thing we’re talking about.
A reader sent me a link to this piece at First Things, a magazine of religion and philosophy that I have to say that I don’t link to very often. It’s a well-written article about scientific reproducibility, and gets a lot of things right (while getting some subtle things wrong, too, in my opinion). By the end, it’s largely an attack, from a religious perspective, on “scientism”, the tendency to treat science as a religion itself. And there the author has a good point. This irritates sincere religious types a great deal, as it should – I’m not religious at all, and it can sure irritate me. Seeing people who know nothing about a particular field (and especially nothing about statistics) citing some journal article to make some political point, and then acting as if no possible counterargument could then ever be made (“You can’t argue with the settled science! It’s peer-reviewed!”) is scientism all the way – received wisdom, taken as true simply because of its (unexamined) source.
I think that the First Things article overstates the problems in science today, but given all the talk about the state of the scientific literature, it’s easy to see how someone could end up doing that. The scientific enterprise, though, is not yet in danger of crashing down under the weight of its own contradictions. Presumptuous human reason is still having its innings, like it or not. The self-correcting nature of science is not some sort of magic sword, that’s for sure, but it’s still a real weapon, and it looks like the best we have. It may well be that out of the crooked timber of humanity, no straight thing was ever made, but science, at its best, is one of the straighter things going.