Skip to main content

Business and Markets

Startups And Their Publications

How much does scientific publication matter? For once, we’re not going to be talking about its role in academia (partly because it obviously means quite a bit there!) No, how much does it matter in industry? Specifically, at highly valued biomedical startups? That’s the subject of this new paper by John Ioannides and co-authors, and I’ll go right to the conclusion: across 47 “unicorns”, startups valued at more than one billion dollars, there seems to be no correlation at all between their valuation and their publication record.

To be honest, I’m not sure if that surprises me much at all. For several reasons. But first, it should be noted that the analysis itself generated a rather lumpy data set. Of the 47 companies (18 current and 29 exited), 8 had no publications at all, and half of the total publications from current firms are from just two companies:

Unicorns published 425 PubMed papers. Only 34 (8%, including two reviews) were highly cited. For exited unicorns, we identified 413 papers, of which 47 (11%, including nine reviews) were highly cited. Overall, more than half of the current unicorns (10/18) and almost 40% of the exited unicorns (12/29) had no highly cited papers. Over the entire cohort of companies, we identified no association between company founding year and number of published (r = −0.09, P = 0.51) or highly cited papers (r = −0.08, P = 0.57).

Three unicorns (Outcome Health, GuaHao and Oscar Health) had no published papers, and two more (Clover Health, Zocdoc) had published just one. All five were in the domain of digital health. Among exited unicorns, five (Enobia Pharma, Neotract, Qualicorp, Cameron Health and China Nuokang Biopharmaceutical) had no published papers and two (Flexus Biosciences and Cardioxyl Pharmaceuticals) had just two. 23andMe (107 articles) and Adaptive Biotechnologies (89 articles) published almost half of all unicorn papers. No similar disproportionality existed for exited unicorns.

The paper extends the analysis to companies below the billion-dollar mark and finds the situation to be (at best) the same, and then comes a line that probably should have led off the whole paper: “Most start‐ups apparently do not publish much“. It’s true. I think it’s been true for a long time. And it’s true for several reasons. Ioannides makes quite a bit out of the Theranos story earlier in the article (stealthy high-valuation company whose tech wouldn’t stand close inspection, etc.) but I think that’s overdone. Companies who have really valuable stuff that’s, you know, real also keep quiet about it until everything is lined up. Not everyone who doesn’t publish is a Theranos.

And even those that do publish aren’t going to have a long record to look at. Remember, these are startups. They haven’t been around for that long, and they’re concentrating on getting stuff to market (or to a stage where they can convince people to do a deal with them towards that), not on writing up manuscripts that will pick up lots of citations. That comes later. Part of this is the disconnect between academia and industry about scientific publication in general. If I came across a busy young company that was spending a lot of time putting papers together for big-time journals when they could be shoring up their science internally, I would not invest. Near the end of the manuscript, this point of view finally makes an appearance:

Publishing is clearly not the primary mission of start‐ups. The need to spend time to write, submit, revise and publish papers may even be seen as a deviation from the trajectory of disruptive innovation. Further disincentive arises from the fact that peer‐reviewers may be resistant to new ideas. Nevertheless, when technologies and products influence real‐world health outcomes, peer‐reviewed publication is essential.

That last sentence may well be true, but it comes across to me more as an assertion than a conclusion. Peer review is overall a good thing, but it does (as mentioned) slap down interesting ideas at times, and it also lets junk through, even into high-end journals. At the same time, there’s no doubt that having other sets of experienced eyes looking over your work is valuable, but there are other ways for that to happen than peer-reviewed publication (and the companies that manage to skip this step in every way, such as Theranos, are indeed asking for trouble). I agree that publication should be an eventual goal for the good of the overall scientific enterprise, but it might asking quite a bit of companies that haven’t even gone public yet, at least in some cases. To its credit, the study does mention that a company doesn’t need a long record of papers – one or two key, detailed ones would be fine. But its count-the-papers approach a few paragraphs before belies that a little.

There’s another feature of the analysis that needs to be brought up: I notice that this work searched “searched PubMed through November 2017 for papers carrying each start‐up’s current or past name(s) as affiliation.” That’s fine as far as it goes, but my guess is that there are a significant number of cases where the enabling technology (or at least its first iteration) was published under a founding scientist’s name but without the company affiliation, because it may well not have even existed at the time. Something to consider, but the paper dismisses this idea by saying that “However, it is unlikely that this work can be considered directly relevant to the start‐up“. I’m not sure about that at all.

What should potential investors take away from this analysis? It probably is reassuring to see what look like good papers in good journals when you’re evaluating a small company – but as Ioannides’ own analyses have made clear in the past (and as anyone with experience in this business knows) that’s no guarantee. In fact, the top-tier journals publish work that is a bit less reliable than the middle-tier journals, in large part because the stuff is so cutting-edge and interesting. So it’s nice, but it doesn’t let you off the hook in the due-diligence department. Patent applications are something you’re going to want to look at, too, but (as the current paper points out) Theranos had lots of patents and applications. Peer review does an imperfect job of catching bullshitters, but the patent office is set up to do that even less.

At the other end, if a company has no publications at all, you really do need to kick the tires more carefully. Big investors will be seeing lots of fascinating PowerPoint presentations and confidential reports, and they’d better be prepared to ask serious questions. There are VC shops who actually pay third parties to try to reproduce exciting results before they’ll invest, when that’s possible. Of course, if you’re looking at a Theranos, the PowerPoints will be full of fiction and the answers to your questions will be slick, reassuring lies. That’s the problem: the scientific enterprise assumes that you are not lying about everything. Peer review (ideally) catches inconsistencies, omissions, and misinterpretations, but you do not start off reviewing a manuscript by asking yourself “What if everything in this paper is a deliberate fabrication?” You’re looking for internal consistency, for adequate proof of what’s claimed, for possibilities the authors may have missed, but you start off by assuming that they’re trying to tell you about something real and not trying to fool you about the whole subject of the paper. That’s how a lot of fabricated stuff slips through, of course (small stuff and large) – we’re not always even thinking about the possibility that that’s what it might be.

What I’m saying is (as above) that referencing Theranos (as this paper does, many times) is a bit of a red herring. You cannot necessarily catch these people by looking at their publication records; it would be a simpler world if that were the case. This latest paper, to my mind, mixes that up with the subject of publication records in general, and it makes for a weaker case overall for what was not a strong argument to start with.

24 comments on “Startups And Their Publications”

  1. Me says:

    Alot of VC investors do require output from the companies, but that manifests as conference talks and citations in news articles rather than peer reviewed publications. I know quite a few people after ‘the scattering’ when my old med. chem. unit broke up that had this issue.

  2. This seems like an odd list of “biotech” start-ups to look at. ZocDoc is a physician identifier and appointment scheduling platform. Is it really that surprising that they had only one published scientific paper?

  3. Myron says:

    Longtime fan of the blog Derek. Think I’ve spotted my first typo.
    “At the same time, there’s no doubt that having other sets of experienced eyes looking over your work is valuable, but but there are other ways for that to happen than peer-reviewed publication”
    Too many but’s. Which is humorous given the phrase that immediately proceeds them.
    Keep the enlightening viewpoints coming,

    1. Derek Lowe says:

      Fixed, thanks! But believe me, there are always plenty more typos where that one came from. . .

      1. gippgig says:

        I just checked and a bunch of the errors I e-mailed you have not been corrected. Am I sending them to the wrong address?

        1. Derek Lowe says:

          Nope – I just have to set aside some time for typo correction! Thanks. . .

  4. b says:

    Another point not considered for lack of publications/patents: trade secrets. Small companies don’t have a team of lawyers to prevent others from brute-forcing their way onto the market before they can. It’s obvious if somebody tries to take your new molecule to market, but if you have a new technique for performing a specific task or synthetic method for making a specific compound, there’s a large debate about whether you even want to patent the process or not (at the early stage). Those are much harder patents to enforce and others can easily swamp your competitive advantage purely by numbers you don’t have ($ and people), and you can’t do a damn thing about it.

  5. Eric says:

    I see two major flaws with this paper. First, publication in peer-reviewed journals is not an indication of the validity of a new technology. Validation comes when it is replicated in other labs. Secondly, there seems to be an assumption that VC are investing blindly in these startups because nothing has been published. VC will typically have seen confidential, internal documents during their due diligence that is not available to the general public (as always, the retail investor is last to know). As Derek points out, this doesn’t prevent outright fraud like Theranos, but it certainly isn’t a blind investment.

    1. JSR says:

      Replication >> Peer review

  6. Hap says:

    1) If a startup is trying to make a platform technology, don’t they not want to publish (to avoid explaining their framework and thus allow others with more money or time to occupy some of their desired space)?
    2) I thought industry mostly cared about products, not publications. If companies want to improve their reputation or try to attract better employees, publications might help, but startups lack sufficient margin in most cases to worry about those things.

    It seems like the purposes of publications are on a skew plane from the purposes a startup is likely concerned with.

  7. a says:

    “There are VC shops who actually pay third parties to try to reproduce exciting results before they’ll invest, when that’s possible. ”

    who are these people, and who do they pay?

    1. JSR says:

      These people are VCs looking to fund (with millions of $$) a professor’s research that says molecule X cures disease Y (#inMice). So the VC pays some CRO to synthesize X and another CRO to administer in mouse model of Y. If totally independent CROs reproduce professor’s result, it’s worth the $$.

      I knew of a “very exciting” project from a famous, multiple startup-founding, centimillionaire professor. Unfortunately the very exciting results could only be obtained by 1 particular postdoc. I think there may be a peer reviewed paper describing the story. Regardless, NewCo was formed (I think with prof’s personal $$ and may a bit from a friendly VC). Without a big startup round, Pfizer scooped up NewCo. But of course it was all BS. Companies that buy early stage stuff really have to take precautions against such flim flammery. Only way to do it is independent replication–and there’s a whole infrastructure of companies to do this.

    2. NJBiologist says:

      In a previous life (as a biologist at a CRO), I got a study request from a VC group: they wanted a replication of a prof’s mouse work before they started looking for investors. A previous round had failed when, at a late stage, prof’s prior mouse work had not replicated. (It went pretty well: we reproduced something like 4 out of his 5 key findings.)

      However, that was the only such request in my several years as a study director. Feel free to interpret it as the exception proving the rule.

  8. Emjeff says:

    In Industry, publications are important, but it is generally at the NDA submission stage onward where they become essential. The effort is usually led by Medical Affairs, whose goal it is to influence prescribers. They are not so concerned about the basic science stuff. But believe me, the clinical stuff has to get out, and there are very hard timelines around those publications.

  9. Isidore says:

    Based on my experience working in pharma and from what I have seen at meetings and in journals and have heard from colleagues what gets published from industry is from projects that are either dead or well past an NDA submission. Since most startups have not reached either milestone their publications are few and far between.

  10. Tom says:

    Would love to see an update on Samumed, now in Phase 3 Trials along with other drugs in different phases.

  11. Anton says:

    The title of Ioannidis 2005 most cited paper was “Why Most Published Research Findings Are False”.
    With that in mind, a number of publications itself cannot be taken as an evidence of anything that we find important about innovation and biotech.
    What is important is that the due diligence process goes deep enough to look at the robustness of evidence (i.e. whether the risks of bias have been properly controlled, data integrity maintained, no selective reporting, etc.).
    And this is exactly where we have to invest time and efforts – still a major unmet need.

  12. JSR says:

    Why do people in industry publish at all? I think it was Derek who once said of J Med Chem papers that they either represented the trash heap or the museum piece.

    1. cynical1 says:

      They publish because their management make it a performance objective. If you don’t publish that year, you have not met your objectives for the year and your performance rating will reflect that, as well.

      There are several reasons for this: 1. Your VP of whatever department you are in gets to stick their name on your publication thus increasing their publications on their resume without doing or contributing a damn thing. (Let’s face it, this is the only way the desk jockeys running the industry can get their name on so many publications). 2. Your boss or VP wants everything neat and tidy to go present your work at their next conference or interview…….and 3. If you haven’t published, they can use that as an easy excuse to lay you off when the next round rolls around. And the next round is always rolling around.

      1. johnnyboy says:

        Yes, I would tend to agree. I’m pre-clinical, and have worked at 2 big pharmas. The first didn’t care much about publications, actually made it fairly hard to do so. The second puts high-impact/high quality publications in its corporate goals, which trickle down to us, the riff-raff charged with actually writing the damn things so that do-nothing management types can pad their CVs without lifting a finger. Ostensibly it’s supposed to be good for the company’s image; in reality my feeling is that they significantly distract from the real business of making drugs. But to be fair, my company is pretty crap at making drugs, so I guess publications is a way to at least produce something…

  13. MoMo says:

    Start-ups have no business publishing academic like papers and I wouldn’t invest a dime in one that did. Secrecy is key while you are spending other peoples money, and I have fired scientists that showed this behavioral phenotype.

    Behave or beware- its your choice. Now get back to work!

    More Molecules- Not More Manuscripts

    1. loupgarous says:

      I’d think that employees of startups also ought to be assumed to have some intrinsic confirmation bias, regardless of their intentions, in publishing on work which might make their authors’ institutions very wealthy in a Big Pharma buyout.

      FDA and the public would be better served if the country paid independent CROs to do safety and efficacy trials across the board. It wouldn’t eliminate the confirmation bias/fraud issue in drug development firms doing their own safety and efficacy work, but it’d sure reduce it.

      1. MoMo says:

        One of my companies got criticized for this- publishing data on clinical efficacy in the NEJM- Fools- Who else is qualified to do this and why should I pay outside hacks?

        Luckily they were corrected through the magic of social media and never heard about it again.

  14. Mark Murcko says:

    Well, in my experience, if you have top-notch scientists in your start-up, and they are doing great science, some of them will want to publish some of that work. It may be for their career advancement, or may be (and the cynics should be sitting down here) because they genuinely want to share what they know to enrich the scientific community. You know, a genuinely altruistic human emotion. (Hope I didn’t just kill off 17% of Derek’s readership.) It makes sense for the startup to support this to the extent possible, for the benefit of those scientists and for the community. In addition, there are also reasons why it may help the startup — to satisfy the current employees, to attract top-notch collaborators and future employees, and to encourage others to share what they know (rising tide that lifts all boats). I think we all know we are generally pretty ignorant, which is why we are so bad at drug discovery, so taking active steps to share information is in the long run beneficial for everyone.

Comments are closed.