There was some Alzheimer’s news the other day, but it wasn’t actually about Alzheimer’s. Not very much. It was more about hype, press releases, and a grievous lack of understanding of statistics.
Via @AndyBiotech and @biotechtoreador on Twitter, I came across this release about an announcement from a small company called Neurotrope, who have been developing bryostatin as a possible Alzheimer’s therapeutic. (Update: a spokesperson for Neurotrope has contacting me, saying that the release I’m quoting from, which was put out by someone called the “Small Cap Investor Group”, has no connection with the company. I have to also note, though, that its author, Sharon di Stefano, specifically states that she spoke with the company’s CSO and CEO about the trial. Her record with small-cap stocks is, to put it mildly, not encouraging). (Update 2: see this article by Adam Feuerstein about di Stefano and how her article ended up being circulated at all. Something odd is going on). Natural products chemists and organic synthesis fans will perk up their ears at that name, because the bryostatins are well known as examples of complex marine natural products. Bryostatin-1 has been reported over the years to have CNS effects, including enhanced memory in animal models, which led to the clinical effort on Alzheimer’s.
But we all know what the clinical success rate is in that field – arguably zero, and inarguably very close to it. Here, though, is what the Small Cap Investor Group flacks had to say about the company’s Phase IIa results:
Its Phase IIb of 147 patients underwent different dosages of Bryostatin along different timelines to see which worked best. Although the study was geared mostly for safety and tolerability, a surprise ensued – efficacy. Even more impressive was the generation of statistically significant results that exceeded FDA standards, which hold that a ‘p-value’, defined as the probability that good results are not due to chance, should be less than 0.1 in Phase II trials. In other words, 90% of patients are responding to a treatment. Neurotrope, on two standardized tests blessed by leading AD organizations, delivered figures even lower, meaning greater probability of results not being accidental.
This is wrong on so many levels. For starters, the definition of a p-value is completely wrong, and so brainlessly wrong that no company should have allowed such rot to go out under their company masthead. Now, p-values get misinterpreted constantly, often by people who should know better, but this is above and beyond. No, make that “below and behind”; that’s more accurate. This number does not mean that 90% of the patients are responding. And to go on, the FDA does not have some standard that a p-value has to be less than 0.1, because most people would start with 0.05 and go down from there if they’re basing decisions on that number. The p-value among patients who completed the therapy was < 0.07. Most statisticians with any dignity refer to values above 0.05 as “not significant”. This is embarrassing stuff.
And the results that Neurotrope “delivered” are being overstated as well, crap about statistical significance aside. Their p-values are one-tailed, meaning that they’re testing for significance while (statistically) ignoring the possibility of failure. This is inappropriate for a investigational drug study, so much so that this tutorial site at UCLA uses the example of testing a drug’s efficacy as a clear example of when you don’t want to use a one-tailed p value. I would suggest that Neurotrope pay close attention to another section of that same FAQ page:
Choosing a one-tailed test for the sole purpose of attaining significance is not appropriate. Choosing a one-tailed test after running a two-tailed test that failed to reject the null hypothesis is not appropriate, no matter how “close” to significant the two-tailed test was. Using statistical tests inappropriately can lead to invalid results that are not replicable and highly questionable–a steep price to pay for a significance star in your results table!
I could not have put it better. Furthermore, the test that the company makes much of is the Severe Impairment Battery (SIB). It’s a 100-point scale, and the company says that the bryostatin-treated patients improved by 2.6 points on it at 13 weeks (difference between treatment group and placebo). Big deal. Aricept (donepezil), an approved Alzheimer’s drug which is already known not to slow the progression of the disease one bit, but merely improve symptoms for a while, has an improvement of 5 SIB points at that same time point (as AndyBiotech noted from the drug’s package insert). 2.6 points is not clinically meaningful – as the company itself shows, it’s only statistically meaningful by the most charitable interpretation possible.
That fluff piece about the company, then, is ridiculous from a scientific standpoint. What am I saying, though – it’s ridiculous from any standpoint. Here, have some more:
“Neurotrope, with scientific vigor, showed positive, significant results from its ‘Completer’ group of patients. . .(bryostatin) acts like a car battery jump-start to grey matter, restoring life and energy to brain components that have died or gone dormant from AD. Early proof from the Phase II study should be viewed with excitement from the AD community.“
How anyone is allowed to go on like this about a publicly traded company is beyond me. “Restoring life and energy to brain components that have died”? I used to hear more believable pitches from Mexican border-blaster AM stations on late-night car rides in the 1970s; this sounds like Reverend Ike selling Prayer Cloths. And at least he admitted up front that he was just in it for the money. <s>What’s Neurotrope’s excuse for this publicity stunt?</s> (Update, as mentioned above, Neurotrope is disavowing any connection to the quotes above – as well they should.)