Skip to main content


A Hard Look At Liquid Biopsies

This new paper has generated a lot of headlines (Science news writeup here). It reports work on the long-sought “liquid biopsy” idea for cancer screening, the use of circulating biomarkers to detect tumors via a blood test. The idea has obvious appeal, so much appeal that many news stories over the years have gotten well ahead of the facts.

Looking closer at this work, it really is the best thing of its kind I’ve seen. And it really isn’t good enough. The team looked at about a thousand patients with clinically diagnosed tumors (stage I to III, nonmetastatic) of eight different types (ovarian, hepatic, stomach, pancreas, esophageal, colorectal, lung, and breast) and tried to detect both protein and DNA markers of their presence. Of these, there are screening techniques available for only two types (lung and colorectal), so something that caught these and others in the general population could be very useful. Hold that thought, though.

The DNA part of the screen uses mutations known in the Catalog of Somatic Mutations in Cancer (COSMIC) database, detected by PCR. The proteins are from a list of 41 reported candidates (39 of which turned out to have some use). The actual test ended up using 16 DNA markers and 8 proteins, because the gain in signal from larger sets was not worth it (or actually made things even noisier), and the combination is called CancerSEEK. Detection varied, as of course it would: ovarian and hepatic tumors had the most sensitivity, with the test picking up over 95% of the tumors in the sample. At the other end of the scale, the breast cancer detection rate was down in the 30% range.

So overall, there were quite a few false negatives. Fortunately, the false positive rate was lower – in a screen of 812 people without any detectable cancer, only 7 showed up positive. That still needs improvement (numbers coming up), but it’s a good start. The bigger problem is if you’re going to use this test for early detection: when the team looked at the detection rates adjusted for the stage of the diagnosed tumors, CancerSEEK turned out to only catch 43% of the Stage I cases overall.

Real-world data is going to be generated on this one – the funding (up to $50 million) is in place for a five-year study in women 65 to 75 who have never been diagnosed with cancer (up to 50,000 patients). The criterion will be two positive readings (because of that 1% false positive rate), after which imaging techniques will be used to try to find the tumors. (I should mention that the team is able to broadly localize the tumors as things stand, from the biomarkers themselves). And here is where the arguing will start.

The key is the actual number of people in that sample who have cancer. With 50,000 seventy-year-olds, you are definitely going to pick up real cases. From these tables of incidence rates, I would guess about 700 people in that sample will actually be diagnosed with cancer (I’ll assume that these are all types that this test will detect – that’s not quite right, but close enough for illustration). So we have 49,300 people who are actually cancer-free. The first pass will tell 493 of them that they have cancer when they don’t, so you’ll test again. That will narrow it down to five people who are actually healthy but have had (unfortunately) two positive liquid biopsies in a row. Meanwhile,  you’ve tested everyone else again, too (48,807 people), and told 488 of them that they have one positive result.

What of the 700 who actually have cancer? The stage at which diagnoses are made varies by cancer type – lung, sadly, is typically picked up at a more advanced point (65% Stage III or IV), while breast cancer diagnoses are 65% Stage I. The hope is, of course, that a simple test like this one will pull more of the harder-to-catch types back to the earlier stages, but as noted above, the problem here is that the detection rate for those early-stage cancers is pretty weak. To ballpark it, let’s say that of the 700 patients in this study that you’d expect to be diagnosed with cancer, that half of them (350) are going to be Stage I. The test is going to miss 57% of them (200 patients) on the first pass, and it’s going to miss another 57% (114) of those people on the second run. Meanwhile, of that other 150 people that showed up positive the first time, 85 of them are going to be missed when you test again. From the paper, the detection rate for later-stage cancers averages about 75%, so of that other 350, you’ll miss 87 the first time around, and 22 of those will be missed on the second pass. Of the 263 later-stage patients who tested positive the first time, you’ll miss 66 of them with an incorrect clean test the second time.

Net among the healthy population (49,300 people) is that you’ve told 48319 of them, correctly, that they’re OK (98%). You’ve given some real worries to 976 of them, though, and flat-out terrified five. Net of the group that actually has cancer (700 people) is that you’ve correctly told 262 of them (37%) that they have cancer. Unfortunately, you’ve also told 136 of them (19%) that they’re apparently OK, because they’ve passed two liquid biopsies in a row. And you have 302 patients (43%) who are in the gray area of one positive, one negative – the same category as those other 976 people who are actually OK.

In fact, since we really don’t know who those 700 patients are, the “blinded” version that we’ll actually see is something like this: out of 50,000 people, 48,455 of them have shown no cancer in two tests in a row (although 136 of them actually have cancer). Meanwhile, 267 people have shown detection of cancer twice in a row (although 5 of them are actually cancer-free). And you have 1278 patients who are one up/one down. 302 of those actually have cancer, although you certainly don’t know which. I should note that these numbers are probably optimistic. The expectation is that the false positive rate will be somewhat higher in the real-world study, for example, since its population is uniformly older.

Update: it’s been pointed out in the comments that I’m assuming here that the false positives are independent each time, which is likely not the case. Inflammation, for example, could well produce them every time you test, so these really are optimistic numbers!

But hold on. We haven’t even gotten to the hard part. Now we get to ask a really tough question: of those 267 people that the study will presumably move to thorough imaging, how many of those cancers should be treated? This is something that not everyone thinks about – normally, the impulse is that if you get a cancer diagnosis that it has to be treated immediately and aggressively. But we know that there are people whose tumors are so slow-growing and benign that they’re going to die with them, not of them. Prostate cancer is the most well-known type with a number of patients in this category. You will do these people a great disservice by giving them surgery, radiation, chemotherapy, what have you, and you will spend a great deal of time, effort, and money that could go to people who need it. As the Science news piece has it:

For those who test positive twice, the next step will be imaging to find the tumor. But that will bring up questions raised by other screening tests. Will the test pick up small tumors that would never grow large enough to cause problems yet will be treated anyway, at unnecessary cost, risk, and anxiety to the patient? Papadopoulos thinks the problem is manageable because an expert team will assess each case. “The issue is not overdiagnosis, but overtreatment,” he says.

That, I have to say, sounds a bit like the NRA’s line that guns don’t kill people, people do. The tricky part will be keeping overdiagnosis from leading to overtreatment, which it generally does. Good luck to the expert team. I have had my disagreements with Vinay Prasad in the past, but this paper in the BMJ is worth reading on this topic. The problem is that the statistics that show that “cancer screening saves lives” are based on disease-specific mortality, not overall mortality. The trials needed to show a benefit in that (unproven!) latter category would have to be huge, but those are the numbers we really need: how many people die? Some types of screening (the PSA test) don’t even seem to show a benefit in disease-specific mortality, much less overall, and the numbers for (say) mammography are very much a matter for debate.

I realize that that sounds flat-out heretical, but the general perception of the benefits of cancer screening really are not in line with reality. That said, a really solid liquid biopsy type test might be the sort of thing that could tip the balance towards unequivocal benefit (after all, there are fewer cases of hepatic or pancreatic cancer that are better left alone). The new work I discussed above, for all its shortcomings, really is an important step towards such a test. But as it stands, it isn’t one itself. That’s for the future, damn it all.

23 comments on “A Hard Look At Liquid Biopsies”

  1. luysii says:

    Quite agree about PSA, a test that has been offered to me many times. I’ve always declined as the evidence for it reducing my chances of dying of prostate cancer (or anything else for that matter) was minimal.

    As noted, the rarer the disease, the better the screening test must be to avoid false positives.

    Not noted are the financial incentives to the physician for doing something for an incidental finding rather than watching and waiting, and the financial DISincentives to the physician for watching and waiting (e.g. malpractice).

  2. Andy says:

    The Cochrane Review on breast cancer screening makes really uncomfortable reading, essentially concluding it does more harm than good.

    1. dearieme says:

      My wife worked on a research project on breast cancer screening: the Principal Investigator came reluctantly to the conclusion that it might well be doing more harm than good. That was nearly forty years ago.

      1. Andy says:

        Plus ca change 🙁

  3. Patrick says:

    Something I believe you left unmentioned but makes the numbers even worse is your math assumes the error rates on repeated tests are independent, but it seems almost certain some or most of it is NOT.

    Many missed or false positive tests are not failing ‘randomly’, they are making consistent, repeatable measurements that simply do not correlate to the desired informational readout. This person has (or doesn’t have) the cancer of interest, but they do (or do not) have the biomarkers (or they have something else that is interacting with the test). Repeating the test will not improve the error rates for these people.

    It’s likely impossible to say what percentage are in that category without very detailed testing of many cases, and I don’t know enough to even ballpark it, but it seems likely to be significant.

    And unfortunately, that doesn’t detract from your point. It just makes it even harder for this sort of test to do good.

    1. gcc says:

      I actually hadn’t read your comment when I wrote mine, but this is exactly what I was wondering about. And now I’m even more curious if there’s data in the paper that addresses this question.

  4. gcc says:

    I’m curious what percentage of the initial false positives would be positive again when tested a second time. Your estimate here seems to be based on the two tests being independent events (so only 1% of 1% falsely test positive twice), but I would guess that a lot more than 1% of the initial false positives would test positive again.

    I don’t have full-text access to the paper, so maybe there’s data in it that supports that idea, but without having read the paper I would guess that at least some of the initial false positives wouldn’t be because of noise in the assay, but rather because of some other non-cancer biological condition that leads to elevated levels of the proteins detected in the test. (Although I guess it depends on how the protein biomarkers and mutated DNA are weighted by the assay… mutated DNA in the blood should be pretty specific for cancer, I presume.)

    1. Imaging guy says:

      You are right. That is what they wrote in the discussion, “several limitations of our study should be acknowledged. ……………. Second, our controls were limited to healthy individuals whereas in a true cancer screening setting, some individuals might have inflammatory or other diseases which could result in a greater proportion of false positive results than observed in our study”. I think DNA mutations are not specific for cancer. KRAS and TP53 mutations (both are used in CancerSeek) are also found in peoples with inflammatory diseases.

  5. sgcox says:

    As I understand, the test has been done on patients already diagnosed with cancer. But that means it is the already advanced disease manifested itself physiologically, right ? This is hardly the earlier detection. In the reported screen of 812 people, did it pick up a single true positive ? That is a real cancer not prior diagnosed ?

    1. Derek Lowe says:

      That’s a good question. I wondered that, too, but in just 812 healthy controls, the chance of picking up an undiagnosed cancer is pretty low. We’ll see what happens when they get into those 50,000 undiagnosed patients over the next few years, though.

    2. Imaging guy says:

      This is what they wrote, “the healthy control cohort consisted of 812 individuals of median age 55 (range 17 to 88) with no known history of cancer, high-grade dysplasia, autoimmune disease, or chronic kidney disease”. I don’t think those who tested positive by CancerSeek did not undergo further tests to determine whether they really had cancer or not because you couldn’t do further tests on just those who were found positive (by CancerSeek), you also had to do these tests on those who were found negative (by CancerSeek). So, in this control group, if you tested positive by CancerSeek, it was false positive by definition.

  6. Another sobering manuscript (this in biorxiv) about early detection and liquid biopsies (H/T @dshaywitz) looks at the theoretical limits of detecting tumor DNA. Based on current knowledge of the amounts of circulating free DNA, the proportion that is actually tumor DNA, and the limits of sequencing technology, the authors come to the conclusion that it may be impossible to detect cancer at an early stage using less than 150-300 mLs of blood (!!) or know a lot more about specific variants that have very high predictive value.

    1. willie x. gluck says:

      Surely, Theranos will soon be able to do the testing with a few drops of blood! 😉

  7. TroyBoy says:

    But yet there are virally associated cancers where liquid biopsies can be sensitive and specific. One example is this study that screened a number of patients for EBV-positive nasopharyngeal carcinoma (NPC), a head and neck cancer prevalent in Southeast Asia. Their results were published here:
    There is an economic consideration though. They screened 20,174 patients to detect 34 early stage NPC patients and this is in a high-risk population. It wouldn’t make much sense to screen a population in the U.S. But if you’re one of the 34 patients, you are grateful because treatment of early stage NPC is quite successful.

  8. Thomas says:

    Maybe a skewed appoach to the customer.
    The future purpose of these test (minimally invasive, just a blood sample) will be to assure the majority of tested people to – most probably – NOT have common cancers. This is the benefit. Those with cancers are usually anyways going to gain some months at best, at the cost of false-positives worries.

  9. Shanedorf says:

    There is still a lot of debate about what to screen for among the various liquid biopsy companies. Some are screening for cell-free DNA, some for circulating tumor cells, some for the rare cancer stem cells. Trovagene developed a method to screen for tumor DNA in urine instead of blood, noting that some tumor DNA passed through the kidney and was easily assessed.
    But what does finding some circulating tumor DNA really tell you ? It could be the result of your immune system killing those cells. It could be DNA from the outer layer of a heterogeneous tumor. Its not likely to be from the cancer stem cells. Our reach exceeds our grasp at this point in development, but glad to see so many working on it.

  10. David Young MD says:

    Another possible role for such a test is in patients who are suspected of having a malignancy just because they are losing weight uncontrollably, have a high familial risk, or have a hard-to-biopsy possible mass seen on a scan. In a smaller cohort of individuals where there is a baseline higher risk, the test may have a better predictive value.

    This is all about Bayesian Theorem, which most people have no clue as to what it means.

  11. Mister B. says:

    It remembers me a case study, when I was student.
    One new machine can detect a terrorist in a million people. (1 ppm), but, it will also give you 5 innocent people, spotted as terrorists.
    Question was simple. Would you use that machine ?

    For one proper terrorist arrested, 5 innocents follow in jail.
    We had to balance benefit / risk and it was a very interesting debate.

    Your example seems to me quite a more delicate subject. I am not familiar with statistics, but what if tests are triplicated ? It is something quite common in the “biological area” and sure it will down uncertainties to an acceptable number, isn’t it ?

    1. snarkybiologist says:

      Spotted the MBA

      Surely the problems from running our systematically flawed tests in duplicate will be resolved by spending more and running them in triplicate.

    2. Derek Lowe says:

      Ah, but every time you run the test you increase the cohort in the middle that has had mixed positive/negative tests. If you test enough times, no one in the whole sample will have a “clean” record either way (!)

  12. AJP says:

    This would be a good lead-in to a treatment of the differences between sensitivity and specificity and how prevalence of the underlying true condition affects positive and negative predictive value.

  13. Anon says:

    Doesn’t the analysis change greatly when additional information is also considered?

    For example, wouldn’t the result from a polygenic cancer risk score, family history and
    past environmental exposures be of significant importance? Would it not seem likely
    that many in the false negative group (i.e., the 136) would actually be identified as
    high risk in a pre-screen? 99.72% of those without cancer were told that they do not
    have cancer. How much closer to 100% will it need to be before for the debate ends
    or people send their sample to China? Other liquid biopsy techniques could be used to reduce the false positive rate (e.g.,amount of ctDNA present).

    Doesn’t the potential of finding a substantial group of patients with early stage cancer provide an opportunity to create a new generation of less invasive and less toxic therapy?
    In such a scenario cancer might truly become a chronic and yet entirely manageable illness.

  14. Anon says:

    How would the numbers change for those in their 50s?
    99.5% of them would not be expected to develop cancer.

Comments are closed.