Here’s a rather testy letter to the editors of The Lancet about some recent work published there by Novo Nordisk and collaborators.
Both trials produce the same finding. . .Each focuses its main conclusion not on this primary outcome, but on one of several secondary measurements: nocturnal hypoglycaemia in the first paper and overall hypoglycaemia in the second. In both, the difference was of marginal significance and no mention is made of adjustment for multiple testing. These lower hypoglycaemia rates in unblinded studies should be considered, at best, hypothesis generating. At worst they are spurious. . .
The Lancet’s reprints are a major source of revenue for the journal, and a major part of drug company marketing. These trials were written and analysed by NovoNordisk statisticians and NovoNordisk-funded professional writers. We applaud their skill, but regret the lack of editorial effort deployed to balance it. . .
“What are editors for?”, asks the letter. This brings up something that we all may have to contend with if the scientific publishing model continues to change and erode. The publishers themselves make much of their status as gatekeepers, citing their coordination of the peer review process and their in-house editing. (The counterarguments are that the peer review is being done by free labor, and not always very effectively, and that the quality of the in-house editing varies from “pretty good” to “surely you jest”).
These papers are a case in point. What if they are, as the letter writers contend, largely just vehicles for marketing? That sort of thing certainly does happen. Will it happen even more under some new scientific publishing system? You’d have to think that the marketing folks are wondering the same thing, but from the standpoint of a feature rather than a bug.
Marketing, though, would rather have papers to point at that are published in a prestigious journal, which is one reason that letter is being sent to The Lancet. And no matter what sort of publishing model comes along, I don’t think that we’re ever going to get rid of prestige as a factor, human nature being what it is. (And beyond that, having a stratum of recognizably prestigious journals does have its uses, although its abuses can outweigh them). It is, in fact, the prestige factor that’s keeping the current system afloat, as far as I can see.
The only thing I can think of to replace it that wouldn’t be as vulnerable to the same abuses would be one where papers float to the top through reader comments and interest. Upvotes, downvotes, number of comments and replies, number of downloads and page views – these might end up as what people point to when they want to show the impact of their papers, along with the traditional measures based on citations in other papers. But while that might avoid some of the current problems, it would be open to new ones, various ways of gaming the system to boost papers beyond where they naturally would end up (and to send rival work down the charts as well?) There’s also the problem that the most-discussed papers aren’t a perfect proxy for the most important ones. A harder-to-comprehend paper, made that way either through its presentation or through its intrinsic subject matter, will make less headway. And deliberately buzzy, controversial stuff will rise faster and higher, even if it’s not so worthwhile on closer inspection.
It’s probably impossible to come up with a system that can’t be gamed or abused. I won’t miss the current one all that much, but we’ll have to be careful not to replace it with something worse.