ScienceInsider

"I think there’s something there," says Eric Hunter of the Emory Vaccine Center in Atlanta, one of 22 scientists who argued in a 2004 editorial in Science that the trial lacked a proper rationale and was a waste of money. Colonel Nelson Michael of the Walter Reed Army Institute of Research, who helped lead the $105 million trial that involved more than 16,000 participants, said at the meeting that their findings are a "yes-we-can-moment."

But the two vaccines used in the study and delivered as a one-two punch protected fewer than one-third of the people who received them, the effects appeared short-lived, and it's unclear exactly how the shots work, or even how much each of the vaccines contributes. "I think for the first time we have a vaccine that has an effect," says Mitchell Warren, who, as the executive director of the AIDS Vaccine Advocacy Coalition, monitors the field closely. "Now we need to find out why, and where we go from here."

The “prime-boost” strategy relied on ALVAC-HIV, a crippled canarypox virus engineered to contain three HIV genes, and AIDSVAX, a genetically engineered version of the HIV surface protein gp120. AIDSVAX alone had failed in two previous efficacy trials, and ALVAC-HIV performed poorly in smaller studies. So when researchers from the U.S. Army and the Thai Ministry of Health first reported that it offered 31.2% protection, many AIDS vaccine researchers were stunned.

But as ScienceInsider reported on 5 October, some researchers who were shown more of the data in confidential briefings complained that the study team had painted a rosy picture in its press announcement by ignoring two analyses that undermined the positive findings.

In today's presentations and in the NEJM paper, the team shows the results of the three analyses. One, called Intention-to-Treat (ITT), contains all 16,402 people originally enrolled in the study. In this analysis, the vaccine had a 26.4% efficacy, but the so-called p-value was 0.08, meaning there's an 8% chance that the results were due to chance, well above the generally accepted 5% cutoff. Another analysis, called Per Protocol (PP), excluded almost 4000 study participants who didn't follow the study design to the letter. (The vast majority missed one or more of the six shots or did not receive them at the appropriate time.) This analysis showed a similar protective effect, but a much higher p-value of 0.16.

The most important analysis, the team argues, is the modified Intention-to-Treat (mITT) analysis, which is identical to ITT, except it excludes seven participants who were discovered during the study to have been infected with HIV before receiving the first shot. This analysis, the only one presented on 24 September, showed the highest efficacy—31.2%—and it was also the only one in which the protective effect was statistically significant, although only marginally so, with a p-value of 0.04.

Focusing on the mITT may look like statistical cherry-picking, but Harvard University biostatistician Victor de Gruttola says this misses the point. “All of the analyses should have been presented initially—they give you a more complete picture of the study—but it’s a mistake to say when you look at three analyses you get a very different picture,” says de Gruttola. "The basic message is it’s a weak signal. It’s not nothing. It’s not compelling.”

Some critics say if the researchers had presented all three analyses on 24 September, they would have avoided much criticism. "If they’d come out with this right from the start, people would have said there’s something interesting here to look over," says John Moore of Weill Cornell Medical College in New York City, another co-signatory of the 2004 Science editorial attacking the study. But Supachai Rerks-Ngarm, a scientist with the Thai Ministry of Health, says the team had promised to present the data to the Thai people first, and it wanted to deliver a clear-cut message. The full analysis "would have been difficult for them to understand," Supachai says.

Researchers hope that careful studies of participants' blood samples may help tease out exactly which immune responses protected people in the vaccine group, information that could be used to design better vaccines. One of the field's major frustrations has been its inability to find so-called correlates of protection, or biological markers for immunity. "This study is now really the only hope we have of finding them," says HIV researcher Joep Lange of the Academic Medical Center in Amsterdam.

The researchers have assembled four teams of experts to advise them on how to proceed with such analyses. And they're going to extraordinary lengths to solicit ideas from colleagues. “We’re in the process of setting up an online submission site so that researchers in the field can submit ideas about how best to use the limited samples available from this trial,” says Mark de Souza of the U.S. Military HIV Research Program based in Bangkok.

A breakdown of the data resulted in several trends that, although not statistically significant, deserve further exploration, the study researchers say. The vaccine appeared most protective during the first year, for instance; after that, new infections in both groups occurred at roughly the same rate. The vaccine also seemed to offer more protection to people who put themselves at low or medium risk of contracting HIV.

Ultimately, Emory’s Hunter says the field may have to stage smaller clinical trials with the same vaccine combination in populations that have higher rates of new infections than these communities in Thailand. “Trying to tease out the correlates of protection is going to be a difficult challenge,” says Hunter.

Given the string of disappointments in their field, AIDS vaccine researchers say they're counting their blessings today. "If this were any other vaccine you’d say these are incredibly disappointing results," says Bruce Walker of the Massachusetts General Hospital in Boston. "Here you see a signal that looks to me like it’s marginal—and that’s exciting."