by Jon Cohen
When the U.S. Army and its collaborators in Thailand announced at press conferences on 24 September that a large clinical trial of an AIDS vaccine had lowered the rate of new HIV infections by about one-third, researchers were surprised and encouraged. Although it was only a modest reduction, it was the first positive result from any AIDS vaccine trial.
Now some researchers who have seen more of the data in confidential briefings are complaining that a fuller analysis undermines even cautious claims of success, and they are raising questions about the way the results were announced.
The press conference and press releases discussed an analysis that included all 16,000 people who participated in the trial, except for seven who were infected before receiving any doses of the two vaccines that were used in combination. Seventy-four people in the placebo arm of the study became infected with HIV, while the similarly sized vaccinated group only had 51 infections—a 31.2% efficacy. The analysis indicated that there was about a 96% level of confidence that the effect was real and not due to chance—just above the 95% cutoff that is widely used as a measure of statistical significance.
In the private briefings, researchers learned that a second analysis, which is usually performed in vaccine studies and was part of the Thai study’s design, also found that vaccine recipients had fewer infections, but the reduction was not statistically significant and the level of efficacy was slightly lower. This analysis eliminated people in both groups who did not rigorously follow the protocols. “Anything that really works, you’ll have enough robustness in results to be significant with both analyses,” says Douglas Richman, an AIDS researcher at the University of California, San Diego, a longtime critic of the study. Richman did not discuss the specific results with Science.
“The press conference was not a scholarly, rigorously honest presentation,” said one leading HIV/AIDS investigator, who like others asked that his name not be used. “It doesn’t meet the standards that have been set for other trials, and it doesn’t fully present the borderline results. It’s wrong.” Two biostatisticians who specialize in HIV prevention trials and have not seen the data, said that the results from all participants are the more important data, but they were puzzled that the press conference did not include the analysis that excluded those who didn’t follow the protocols. “I think if people saw [the two analyses] diverging in a vaccine study, they’d have a lot of questions,” says David Glidden, a biostatician at the University of California, San Francisco.
Colonels Nelson Michael and Jerome Kim, researchers with the U.S. Army who helped run the study, strongly objected to the assertion that they gave the data a positive spin. “We needed to get enough information out so we could give the world community a fair glimpse of what we’ve learned,” said Michael. He notes that in addition to showing select researchers a fuller presentation of the data, a paper under review at the New England Journal of Medicine describes both analyses, and all the findings will be discussed on 20 October at an open AIDS vaccine meeting in Paris. “We tried very carefully to make sure that message was crystal clear,” said Michael. “There’s now hope. But that said, we’ve tried to be very careful not to oversell this.”
Several researchers wonder why the data were even released publicly before the Paris meeting. People running the trial learned the results on 10 September, and Michael said there was concern that the information would leak before 20 October. Thai collaborators asked for the 24 September date, Mahidol Day, which commemorates the passing of the current king’s father, a clinician who studied public health at Harvard University.
The debate over the way the results were presented will have no immediate practical impact because even under the most optimistic assessment, the vaccine offered too little protection to be a serious candidate for widespread use. But a modest success, even one that is marginally significant, may point the way to new vaccine strategies. “I think that the field is energized,” said Kim. “Everyone should wait until the data are out.”
Several critics point out that a press conference 2 years ago about another AIDS vaccine efficacy trial—which was prematurely stopped because the product clearly was not working—researchers reported both analyses (subs. req). What is more, the AIDS Vaccine Advocacy Coalition, a nonprofit that issued a report about interpreting the Thai results before the data were unveiled at the press conference, stressed the importance of both. “The safest route is to report both ... and to analyze any difference,” the report states. “Advocates’ take-home message: no matter what the headlines say, a single number is not the full result.”
@don,,,,may be they are wasting millions, but do you have any alternatives: trying Phase III trial without spending money. I do agree that data are misinterpreted, so we should be more focused on this.
People, people: calm down! The little kernel of hope you can take away from this is that heterologous prime-boost vaccine regimens MAY work a whole lot better than the other regimes that have been tried so far, EVEN with lousy vaccines!!
Think about it: two vaccines which are not very good, in combination - appear to have SOME efficacy. Imagine that. Think of what two BETTER vaccines - as long as they are different - might do.
No heads should roll. No-one should resign. What SHOULD happen is that some of the pundits ought to eat humble pie, stop prognosticating, and actually get some vaccines out there and into trial.
Thai scientists are well capable of running a correct trial. Not third world country which needs to announce results on holiday.
Please follow on with new story with names of critical scientists.
This study was powered for intention to treat model
http://www.hivresearch.org/phase3/phase3update-20091010.html
http://www.primeboost3.org/eng/index.php?option=com_content&task=view&id=26&Itemid=37
This is going to hurt the entire AIDS vaccine effort. What are people going to think? Does the AIDS vaccine research community have any clue what they are doing?
The NIH hierarchy and other prominent leaders were quick to praise the "positive" findings released from the Thai trial. They should be ashamed. Differences between the vaccine and placebo groups are marginal at best, if significant at all. Unfortunately, there is a strong cohesive need among this group to justify the hundreds of millions of dollars that they are flushing down the toilet in wasteful human trials. And look carefully at Jon Cohen's article regarding the comments of scientists in the field. "who like others asked that his name not be used". "did not want to discuss the results in detail". Scientists who depend on NIH for their recognition and funding are afraid to speak up, are afraid to rock the boat of the hierarchy. The change in direction promised by the NIH hierarchy at "the summit" is a crock of s--t
Ridiculous. The press announcement by the leaders of the Thai trial was deceptive and grossly inappropriate. Heads should roll.