John Ioannidis is back with an article titled “Why Most Clinical Research Is Not Useful”. (Thanks to Cambridge MedChem Consulting for the mention of it). His emphasis here on clinical work comes from his own admission that improving the efficiency of early-stage research is much harder to do, since it can lead in so many directions that aren’t apparent at the start, but that clinical research should be much more focused and specific. From the title, at least we know that his delicate and tactful approach remains intact. A less incendiary choice might have been “Why Most Clinical Research Isn’t As Useful As It Should Be”. But even if you find Ioannidis annoying, he tends to make good points, so what does he have to say here? First off, some definitions:
The term “clinical research” is meant to cover all types of investigation that address questions on the treatment, prevention, diagnosis/screening, or prognosis of disease or enhancement and maintenance of health. Experimental intervention studies (clinical trials) are the major design intended to answer such questions, but observational studies may also offer relevant evidence. “Useful clinical research” means that it can lead to a favorable change in decision making (when changes in benefits, harms, cost, and any other impact are considered) either by itself or when integrated with other studies and evidence in systematic reviews, meta-analyses, decision analyses, and guidelines.
Fair enough. He’s suggesting a series of questions to be asked about any given clinical research paper. Parapharasing, they are: (1) Does it address a problem that’s worth addressing, in its impact on human health? (2) Has the prior evidence been studied to make sure that this new work is adding something? (3) Is the study powered enough to provide useful information? (4) Is the study run under real-world conditions, or as close as possible? (5) Does it fit with what patients find important as well? (6) Does it provide value for the money? (7) Is the goal of the study feasible in the first place? (8) Are all the methods and data provided clearly and without bias?
I might have put those in a different order, but they’re all good questions, and many of them are rather high bars to clear. Let’s apply them to the most recent clinical data discussed here on this blog, the announcement by Sage Therapeutics yesterday. Here’s how I’d answer the Ioannidis questions in that case: (1) Yes. Postpartum depression is a real problem, and can be life-threateningly severe in some cases. (2) Very little is effective in such cases, so this is fine. (3) No. Not at all. A major failing, which could well be enough to rot the rest of this particular barrel of apples. (4) Not enough details here, but I believe that the depression measurement scales being used are reasonably applicable (and I don’t know of anything better). (5) Most definitely. (6) Hard to say. It certainly was an inexpensive depression trial, as these things go, and thus may well have provided plenty of value, but see the third question. (7) It should be. Post-partum depression is a hard therapeutic area, but there’s nothing that says it’s intractable. (8) Since this isn’t a published paper (yet), we’ll have to reserve judgment.
With that third question example in mind, I’ll provide my own counterpoint to the Ioannidis list, in a set of related devil’s-advocate questions that will, I think, show what he’s trying to have everyone avoid. In the same order, they are (1) Has this study invented a disease that’s actually not something anyone is worried about? (2) Does it ignore previously reported effective treatments and pretend that there’s nothing else available? (3) Does it use so few patients, or for so short a time, that no one can really be sure if anything worked or not? (4) Does it use surrogate endpoints to make it look as if things worked, or pick and choose among outcomes to get a nice-looking result? (5) Does it solve a problem that no patients really wanted solved? (6) Does it spend a vast amount of money to advance the science a couple of inches? (7) Did it start because everyone was being too optimistic about this stuff working at all? (8) Are the raw data stuffed in a box, so that you’ll just have to take the paper’s word for the statistical workup? It will not be difficult to find clinical studies that violate one or more of these. Not at all.
Ioannidis has a section on the complications of applying these criteria, because there certainly are some. For example, he adduces one of his own first papers, on zidovudine monotherapy. When the study was started, it was a relevant question. When it finished, it was still of interest. By the time it was published, though, it was a moot point. For these and other honest reasons, I think it really is too much to expect that all the clinical papers that appear will always get a clear “yes” to all eight questions, but I think he’s right that the current situation is too far over to the other side of the scale. There really is a lot of junk out there.
It would be of great interest if every clinical report actually had to have a little section addressing a standardized list of such questions. It would, in one sense, be an invitation to boilerplate, which is what happens in research grants and other such forms, but at least we could see up front when people were being disingenuous in that way. “Of course our study is relevant, twitchy hair follicles are a major public health problem and our six-patient study advances the field greatly”. In many cases, if you’re willing to run a crappy study in the first place, you’re probably willing to sell it as something better-looking, too. The utility of this question format might come in when people don’t necessarily realize, or haven’t quite been able to admit to themselves, that their study is (in one way or another) lacking, and thinking about this up front, with a realization that it will affect the eventual publication, might be of some benefit.
That last motivation is especially important for academics. For industrial clinical research, the big motivator is What the FDA Will Think, and in some cases that’s one of those complications mentioned above. There are studies that can be expensive and may not (from an outside perspective) move the science quite enough for the money, but are nonetheless run because the FDA says to run them. To be fair, companies often end up in that position when the earlier studies weren’t designed well enough to give data that would have given them a chance to stand on their own. It’s going to be hard to outlaw wishful thinking.