Not too long ago, I was talking to someone outside the field about the “reproducibility crisis”. They’d heard that there were many published papers whose results weren’t solid, and wanted to know if I’d encountered that. I had to tell them that yep, I sure had, and that just about anyone who’s worked in any field of science will be able to say the same thing.
But, I cautioned them, that doesn’t mean the whole scientific enterprise is about to collapse. Less-than-reliable papers have been a feature of it since the beginning. I’m very willing to put money down on the proposition that the first Fellows of the Royal Society grumbled to each other in private about that latest correspondence from so-and-so, which had those things in it that no one else seems to be able to get to work. It comes with the territory, and the (more or less built-in) error correcting mechanisms of scientific work take care of many of them. It can be intensely annoying to be part of that error-correcting machinery when you were intending to do something else with your time, but that’s the risk you take.
I told my questioner that wonky results are found in all sorts of journals, but the pattern was not what they expected. Sure, I said, there’s junk in the junk journals. I mean, some of that stuff is real and published by people who don’t know any better, are taking the easy way out to rack up publications for some committee or government oversight, or just can’t seem to get their stuff published anywhere else. But there’s plenty of actual slop down there, and surely some actual fraud, too – it’s just that no one, for the most part, pays attention or ever tries to reproduce the stuff, because so much of it is of no interest to anyone (including, I’d also be willing to bet, the authors themselves in some cases).
Then, I told them, you have a solid layer of solid journals. For organic chemistry, I’d call this the “JOC stratum” – perfectly respectable journals publishing perfectly respectable papers. It’s not always the most exciting stuff in the world, but it’s almost always reliable, and when you get a literature reference from such work you feel pretty confident that you can get it to run for you, too. Most scientific work is actually in this category, fortunately for us. You don’t see as much real fraud in this part of the literature, for the same reason that no one goes to the trouble of counterfeiting $10 bills.
But after that, you come to the big, flashy stuff. I explained how each field has its own top-tier specialty journals, and after that come the multidisciplinary ones covering (say) all of chemistry (like JACS or Angewandte Chemie) or crossing fields entirely (like Science or Nature). In theory, papers published in the latter journals are of interest to people in totally different fields, although in practice it’s not like we can understand the fine details in each other’s work, and we’ll will do well if we even look over the list of titles. But certainly, if there’s some big to-do in physics or evolutionary biology, I enjoy hearing about it and reading up on it to the extent I can.
Those journals, though, I went on to say, are where the unreliable results start cropping up again. My interlocutor was surprised at that one; he thought that the top journals would necessarily publish the most rock-solid stuff. Not so, I told him – the top journals are such desirable places to publish that people who really need to get papers in there often rush them out before the cake is quite baked. Or before they check to make sure that they added the raspberry jam inside it. Or that they didn’t actually use a half cup of salt instead of a half cup of sugar – you know, that sort of thing. And besides, the kinds of results that can make it into these journals are the high-profile, unexpected, cutting-edge stuff, and those are the tricky things that aren’t always going to work quite right, anyway, not until more people have had a chance to find out the things that can go wrong. Which is one of the big things that publication is supposed to do (other than help your grant renewals or get you tenure).
So there’s reproducibility and there’s reproducibility. At the low end, you have the stuff that Pauli used to describe as “Not even wrong”. At the high end, you occasionally get the wax melted off your wings as you fly too near the blazing sun of a big discovery. And while there is fraud, and while fraud needs to be discovered and expunged (and its practitioners removed from the field), at the low end it doesn’t have a chance to do much damage, and at the high end it tends to get discovered pretty quickly. And actual fraud up at that level is more rare than it is in the low end, for just that reason, which makes the cases where it does happen (as in various retracted stem cell work, or the physics papers of Jan Hendrik Schoen so notable. Those things waste time and effort when they do happen, because so many people are interested in the results, but it’s also true that getting them out of the literature as quickly as possible is not really a waste of time. You’d rather never see such things take place at all, but we staff the labs with humans, and any large sample of people is going to include some interesting and fraught personalities in it. . .