Skip to main content

Alzheimer's Disease

Checking The Numbers on the Alzheimer’s Test

The news of a possible diagnostic test for Alzheimer’s disease is very interesting, although there’s always room to wonder about the utility of a diagnosis of a disease for which there is little effective therapy. The sample size for this study is smaller than I’d like to see, but the protein markers that they’re finding seem pretty plausible, and I’m sure that many of them will turn out to have some association with the disease.
But let’s run some numbers. The test was 91% accurate when run on stored blood samples of people who were later checked for development of Alzheimer’s, which compared to the existing techniques is pretty good. Is it good enough for a diagnostic test, though? We’ll concentrate on the younger elderly, who would be most in the market for this test.The NIH estimates that about 5% of people from 65 to 74 have AD. According to the Census Bureau (pdf), we had 17.3 million people between those ages in 2000, and that’s expected to grow to almost 38 million in 2030. Let’s call it 20 million as a nice round number.
What if all 20 million had been tested with this new method? We’ll break that down into the two groups – the 1 million who are really going to get the disease and the 19 million who aren’t. When that latter group gets their results back, 17,290,000 people are going to be told, correctly, that they don’t seem to be on track to get Alzheimer’s. Unfortunately, because of that 91% accuracy rate, 1,710,000 people are going to be told, incorrectly, that they are. You can guess what this will do for their peace of mind. Note, also, that almost twice as many people have just been wrongly told that they’re getting Alzheimer’s than the total number of people who really will.
Meanwhile, the million people who really are in trouble are opening their envelopes, and 910,000 of them are getting the bad news. But 90,000 of them are being told, incorrectly, that they’re in good shape, and are in for a cruel time of it in the coming years.
The people who got the hard news are likely to want to know if that’s real or not, and many of them will take the test again just to be sure. But that’s not going to help; in fact, it’ll confuse things even more. If that whole cohort of 1.7 million people who were wrongly diagnosed as being at risk get re-tested, about 1.556 million of them will get a clean test this time. Now they have a dilemma – they’ve got one up and one down, and which one do you believe? Meanwhile, nearly 154,000 of them will get a second wrong diagnosis, and will be more sure than ever that they’re on the list for Alzheimer’s.
Meanwhile, if that list of 910,000 people who were correctly diagnosed as being at risk get re-tested, 828 thousand of them will hear the bad news again and will (correctly) assume that they’re in trouble. But we’ve just added to the mixed-diagnosis crowd, because almost 82,000 people will be incorrectly given a clean result and won’t know what to believe.
I’ll assume that the people who got the clean test the first time will not be motivated to check again. So after two rounds of testing, we have 17.3 million people who’ve been correctly given a clean ticket, and 828,000 who’ve been correctly been given the red flag. But we also have 154,000 people who aren’t going to get the disease but have been told twice that they will, 90,000 people who are going to get it but have been told that they aren’t, and over 1.6 million people who have been through a blender and don’t know anything more than when they started.
Sad but true: 91% is just not good enough for a diagnostic test. And getting back to that key point in the first paragraph, would 100% be enough for a disease that we can’t do anything about? Wait for an effective therapy, is my advice, and for a better test.
Update: See the comments for more, because there’s more to it than this. For one thing, are the false positive and false negative rates for this test the same? (That’ll naturally make a big difference). And how about differential diagnosis, using other tests to rule out similar conditions? On the should-you-know question, what about the financial and estate planning implications of a positive test – shouldn’t those be worth something? (And there’s another topic that no one’s brought up yet: suicide, which you’d have to think would be statistically noticeable. . .)

21 comments on “Checking The Numbers on the Alzheimer’s Test”

  1. Michael says:

    This assumes the existing methods are 100% accurate which is far from a given, no?
    Could it not be fairly easily be the case that the new test is 99% accurate, but the existing ‘by hand’ diagnosis is only 90% accurate leading (some of) to the observed discrepency?

  2. John Johnson says:

    Gotta push back on this one. Having a test like this (as we do for Huntington’s which, unlike this one, is about as definitive as you can get) enables clinical trials for the prevention/early mitigation of AD.
    Granted, hopefully there will be another diagnostic test which can be used in combination with this one to provide more specific results, much like they’ve been doing for years in HIV. A good AD testing system will, of course, improve the quality of early intervention trials.
    On the flip side, there is the potential for suicide for people who test positive, which would be doubly tragic for the false positives. We definitely have to approach diagnostics for neurodegeneratives very carefully.
    Overall, I think that even at this stage of the game advancing the diagnostics will be good for research, and hopefully applied research in the next few years.

  3. Chris says:

    You are A) erroneously assuming that specificity and sensitivity are the same, and are what they mean by “accuracy,” and B) probably incorrectly assuming that the problems with the test are probabilistic, rather than systematic (repeated tests will get the same result).
    If they were probabilistic, and the sensitivity=specificity=91%, it would be very, very easy to simply have everyone take ten or twenty tests, resulting the chance of an incorrect diagnosis being less than the chance that the person will get sideswiped by a tractor-trailer on the way to the testing center.
    And as a person above noted, the fact is, psychology is still a very soft science. There is a reason that there is no consensus on whether ADD is one, two, seven, a thousand, or 0 diseases (some say it’s merely a personality subtype). There is a reason we call it “the autistic spectrum,” rather than “autism.” There is a reason that prosopagnosia, the utter inability to recognise other people’s faces, was only acknowledged as a common disorder, rather than a unique condition, with the advent of the search engine.
    It’s entirely possible that the error rate of the human examiner in finding signs of a distinct physical disorder of varying intensity and progression is greater than the test.

  4. PG says:

    I would argue that any *accurate* diagnosis is useful even if there is currently no effective therapy. In the case of diseases like Alzheimer, which has such a long-term burder (physical, emotional, and financial) on spouses, families, and friends, a person diagnosed with a very high risk of getting it may be able to make choices (like saving money for future care) that he/she might not have made otherwise.

  5. Anonymous says:

    91% isn’t nearly good enough for a front line, mass population, diagnostic. But, if it’s given when there are other symptoms presenting, then it’s OK at best. It is also very important to distinguish between false positives and false negatives. I have 1st hand experience with this: cancer screen is 90% accurate, with only false positives (false negatives are very rare). You can be clean, but 1 out of 10 times it’ll turn up positive. Then you just get a CAT scan and redo the test. Pain in the butt? Expensive? Yes and yes. Peace of mind knowing you’re clean? Priceless.
    In the Alz screen (link?), are the false positives and negatives evenly distributed, or is one more prevalent? This is an important distinction. In your example of 20M people, if it only gives false positives, the test is 100% accurate for the 1M that will get Alz; there are no 91,000 being told they are OK. The 19M clean population is still getting hosed.

  6. A-non-y-mous says:

    Sorry, I didn’t give my “name” to my comment above.

  7. Repeating the test may give the same 91/9 split to the same people no matter how many times they take it if there is an uncontrolled variable floating around.
    It reminds me of drug testing athletes. The initial urine or blood draw is split into an A and B. If A is positive, then the B is also tested by the same lab using the same equipment. Is it any wonder that the B test result is always the same as the A test result?

  8. HelicalZz says:

    Is 91% good enough. Depends on the test and what it is used for. If it is simply a cheap screening then yes it is probably good enough.
    For example, lets say I have an injectable reagent which combined with a brain CT scan is 99% accurate (and these are in development for Alzheimer’s), but it costs $2K. I also have a protein chip or test similar to the one discussed that costs $150, but is only 91% accurate. Then the first test should certainly be used as a justification for the second more expensive test (simple economics).
    So in general, 91% isn’t always good enough, but sometimes it is.

  9. qetzal says:

    Some numbers from the paper:
    The 91% comes from testing a group of 47 people who were first diagnosed as having mild cognitive impairment (MCI). They gave blood samples at the start, and then were followed up over 2-6 years. During that time, 22 developed “confirmed” Alzheimer’s (AD), 8 developed other dementias (OD), and 17 showed no progression.
    When the test was run on the time 0 blood samples, it correctly predicted 20 of the 22 patients who developed AD (20/22 = 91% true positives; 2/22 = 9% false negatives).
    None of the 8 people who developed OD tested positive at time 0. However, 7 of 17 people who did not develop AD or OD tested positive. If you combine these two groups to get all people who did not develop AD, the false positive rate was 7/25, or 28%.
    That’s pretty high.
    It’s also important to consider the tested population. These weren’t random older people. These were people who were already diagnosed with MCI. They had a much higher risk of AD than the general population. 43% (20/47) developed AD within 6 years! And even then, the test had a very high false positive rate.
    Hard to say what might happen if a healthy population was tested. Maybe the false positive rate would be much lower. For example, some of the above false positives might not be false. Follow-up was only ~ 6 years, so maybe some or all of those 7 will convert to AD later.
    OTOH, with a 28% false positive rate even among people who are like to develop AD, I wouldn’t be surprised to see a similarly high rate among healthy subjects.
    Don’t get me wrong. I think these results are extremely promising from a research perspective. But it seems clear that they still fall short of having general prognostic value.
    P.S. I agree with John Spevacek. I doubt the false positives and negatives represent random errors. More likely, they are correctly measuring whether the blood markers do or do not fit the ‘AD profile.’ Some people will almost certainly have that profile, but never actually develop AD. Others will not have the profile, but will develop AD. You could repeat the same test on them as much as you like, and always get the same result, without changing your true ability to predict disease.

  10. qetzal says:

    The statistics of diagnostic testing in generally healthy populations can be a bit counterintuitive. Let’s take Derek’s number of 20 million elderly with a 5% chance of developing AD. Let’s further assume we have a test that predicts AD with a 10% chance of false positives and a 10% chance of false negatives.
    We test all 20 million. Of the 1 million that will go on to develop AD, 900K will test positive (true positives) and 100K will test negative (false negatives). Of the 19 million that will not develop AD, 17.1 million will test negative (true negatives) and 1.9 million will test positive (false positives).
    Now imagine you’re one of those 20 million. Before you take the test, you know you have a 5% chance of developing AD. You take the test and get a positive result. What does that mean? Either you’re one of the 900K true positives, or one of the 1.9 million false positives. So now your chance of developing AD is 32%. Your odds of developing AD went from 1:20 to 1:3.
    In other words, even though the test gives the correct result 90% of the time, we still can’t say which people are likely (> 50% chance) to develop AD.

  11. Anonymous says:

    How can you run a clinical trial on a potential Alzheimer’s treatment without knowing who has it and who doesn’t? So yes, a diagnostic test would be a great thing to have even without a cure/treatment currently, and I stress currently, available.

  12. Klug says:

    Could someone post numbers on what a gold standard medical test is? I would imagine that it’s a lot higher than 91%, but I don’t know.

  13. Chris P says:

    If the autopsy data shows the signs of Alzheimer’s disease (plaques and tangles) but the clinical signs were not manifested pre-mortem, then did the person have Alzheimer’s disease or were they in a pro-dromal state that we call normal? Those of us working in this area do not have a flawless biomarker (like cholesterol or blood pressure) to measure to see if someone is at risk for an event (heart attack, stroke, death). Some have pointed to this problem when trying to explain their trial results (a mixed population). I believe clinical diagnosis rate is only in the high 80s for sensitivity and specificity compared to the post-mortem data. This is one reason why none of the guidelines ask for a dementia screen.
    In the Facing Dementia Survey 35% of physicians believed that specialists have a hard time diagnosing AD.

  14. Anonymous says:

    I am not sure what the value of this test would be, and why any person would take this test. The social and ethical issues with this type of test are huge. The lack of value (prognostic, actionable value) for a physician would prevent the reimbursement by the insurers, so it would be an out-of-pocket test. Who in their right mind would plunk down a bunch of money to get a piece of paper that says, “Hey, you are going to get AD! Maybe. Or Maybe not! Either way, do not tell your insurance company about this test or they will exclude AD treatments as it was a pre-existing or pre-ordained condition. That will be $5000 please.” On the other hand, if the information from these proteins can be refined, there may be treatments in the future. Have they looked at the RNA? I am not sure that pure protein screening is the answer, but a lot of smart people are looking in that direction.

  15. Dr. Hetzel says:

    Yes, the ‘comments’ get to the meat, and thus are more useful than the blog itself.
    ‘Update’ should be entitled, ‘Errata’.

  16. WonderDrug says:

    I don’t see what the big deal is here. There are risks associated with every test. Notifying the patient and making sure he understands the probabilities is important. There is no reason to discount the test which is still better than no test at all.

  17. Derek Lowe says:

    Dr. H, the entry barrier to starting a blog is as low as can be. Start one, and I’ll link to it.

  18. leslie graham says:

    is there an actual connection between proton pump inhibiters and alzheimers?

  19. leslie graham says:

    is there an actual connection between proton pump inhibiters and alzheimers? i have never posted before so why does this system say that i have posted too many times?

  20. David VanLandingham says:

    There are excellent reasons for an early diagnosis of Alzheimer’s. Current drugs slow the progress, drugs in Phase III trials have the potential to stop the progress and perhaps even reverse some of the damage. An early diagnosis allows a person to take meds that help them retain their abilities until the drugs in Phase III trials come onto the market.
    Additionally, an early diagnosis allows the patient to make financial, medical and legal determinations before they are deemed incompetent.

  21. rc says:

    Your hypothesis is severely flawed. You do not test all 20 million people, of course that would yield a large number of false positives no matter how accurate the test.
    You only test the patients where there is a reason to suspect they have the disease. The rate of false positives drops dramatically under this paradigm.
    You know better, not sure why you proposed testing an entire population to illustrate your point. You could make even the best test look bad under those circumstances.

Comments are closed.