Skip to main content

Academia (vs. Industry)

The Hype Problem

This paper is right, and it says things that need to be said. I wonder, though, if saying them will do any good. Let me explain what the heck I’m talking about! The paper is titled “Blocking the Hype-Hypocrisy-Falsification-Fakery Pathway is Needed to Safeguard Science“, and I don’t see how anyone can really disagree.

When do the exuberant joy human beings naturally feel in a discovery, and the elation they experience in communicating it to others, overstep the boundaries of acceptability and transmute into falsification of process, evidence or conclusions? How can hypocrisy by scientists and their key institutions be exposed and discouraged, to prevent degeneration of the entire system into fakery? Finally, who can and should take responsibility for ensuring probity in behaviour? These questions concern the foundations of trust that are essential to the pursuit of science and to sustaining the support and attention that society gives to scientists, their work and findings.

Tough questions. My own take is that one could start by targeting the worst excesses, and then work down to the borderline cases. The worst behavior, to me, is deliberate, conscious hype: doing it when you know that you’re coming close to the line or stepping over it. Exuberance is one thing: excitement and pride in your own work can cause you to say things about it that can’t be backed up. But that’s different from sitting down and saying “All right, how can we generate the biggest headlines?” Because that leads to headlines about how you’ve cured Alzheimer’s disease, and believe me, you probably haven’t.

You see this sort of publicity-mongering from all players, academia and industry. University press offices are notorious, not least because the people writing the press releases sometimes have only a vague idea of what they’re writing about or its context. “Whassamatta U. Scientists Announce Breakthrough” is the usual headline form, and the usual reaction is “Yeah, right”. But God knows there’s a geyser of hype coming from industrial sources as well. Small biopharma companies basically live off it, because they’re competing for funding in a crowded, noisy environment and all feel like they have to shout and jump up and down in order to be noticed at all. Larger companies are not immune, either.

The intersection of this problem with scientific publishing is perhaps the most glaring example, though:

This pressure was building throughout the 20th century as science expanded, became better funded and increasingly competitive at all levels. It was boosted by the advent of the digital age and by the growing influence of commercial publishers, with intensifying competition among commercial and learned society publishers for a share of the lucrative, expanding market for science journals. Further pressure came from increasing competition between academic centres to enhance their reputations and income by efforts to recruit and retain leading scientists. As discussed in a recent analysis,  the intersection of the science advancement system (via publications), the career reward system (status, promotions, grants and prizes) and the financial system (large profits from publishing journals) has created perverse incentives for authors, publishers (both commercial and learned societies) and institutions to game the publishing system to their own advantage.

And that one, unfortunately, is mostly an academic problem, because industrial scientists generally don’t care as much about publications. Some companies add in publication as an ingredient in performance reviews, but even then it tends to be a minor one. But in academia, it is the undisputed coin of the realm and the competition is fearsome. Generating papers and getting them into the best journals possible is what many academic scientists would describe (for better or worse) as their job or a large part of it, if they are required to be honest about the situation.

The authors of the current paper go into a number of aspects of the problem – from catchy but uninformative cartoon graphical abstracts, through over-claiming generality of methods in papers, to breathless press releases and on down to the darkness of predatory journals and outright fakery. It’s not a pretty sight – I’ve written many blog posts about these issues and others, and I think every single working scientist immediately recognizes the problems. (The authors mention this as well:

In personal conversations with colleagues, a majority agree strongly with this criticism, but there is general unwillingness to complain loudly for fear of being penalised. In the past, key actors in the science system, including the publishers, the funders and the numerous committees involved in evaluations for grants, promotions, university rankings and prizes, have either remained indifferent or acquiesced in silence and often contributed to weakening the restraints.

“Weakening the restraints” is a general criticism of the current era, if you ask me, but let’s not get into that. Let’s start, then, by continuing to acknowledge that the current system of science has problems, that those problems have become worse with time, and that this has happened because of incentives for it to happen. And then let’s look at those incentives, shall we? Because that’s where change will come from. Telling people not to act in what they see as their self-interest is largely futile. Changing their minds about what their self-interest should be is very much a long-term project. But making it less easy to do the wrong thing, while sometimes a dangerous expedient if taken too far, is often a more immediately available solution and is still the place to start. . .

38 comments on “The Hype Problem”

  1. John Wayne says:

    Unfortunately, the ‘Hype-Bust-Reality’ is important for industrial scientists because riding the ‘Hype’ train to the top is great for your career.

    1. Hap says:

      Failing big (as long as the consequences are passed along to others) is supposed to be an effective path to advancement in management. It seems obvious what that rewards and what it should lead to, but….

    2. Unfortunately, That is totally true.

  2. Peter Kenny says:

    Could this be an opportune moment to remember: Project ranks billions of drug interactions predicts mechanisms through computation (linked as URL for this comment) in which one of quoted sources gushed that “It’s the largest computational docking ever done by mankind”? It should be that this article was written by Nature staff writer and, even six years later, it still makes me giggle.

    1. Dominic Ryan says:

      I note that ‘’, the website that was ‘in testing’ at the time of that Nature paper in 2013, is now a domain for sale: reality, that ugly task master.

      That makes me curious about hype-dispair-useful cycle we have all seen. Has the cycle time shrunk in the last 10 years?

      1. Peter Kenny says:

        Don’t knock it, Dominic, since a site that once featured in Nature can be yours for only 24 monthly payments of $225.

    2. Wavefunction says:

      Number of compounds docked or virtually screened is a vanity metric, so is computational firepower, number of GPUs etc. The only thing that matters is the number of *experimentally validated* true positives with validated target-specific binding, ideally with good PK/PD and novel chemical matter. Otherwise it’s little more than a liquid dispensing contest. Sadly, the current cycle of hype is nothing more than a manifestation of “It is difficult to get a man to understand something, when his salary depends upon his not understanding it.” It’s actually worse than that because many of the people involved in perpetuating the hype actually do understand, and they should know better.

      1. Academic Chemist says:

        I love this comment.

      2. anon says:

        Also in other fields of science there’s very often a huge amount of hype about CPU/GPU cycles that has been used in some simulations as if it were some kind of indicator of important (or even correct) results. Nature journals in particular seem to be very susceptible to this kind of hype.

      3. How many times must I tell you, Ash, that you need stop reading In The Pipeline so that you can rid yourself of this negativity. I’m sure the folk in HR (with their synchronized swimmer smiles) can send you on a course to make you a more worthwhile human being although I should admit, from personal experience, that these don’t work in the more refractory cases. While making comments like “liquid dispensing contest” may seem witty (I do have to admit that I almost wet myself) you really need to accept that if people are sufficiently smart then they no longer need to account for numbers of parameters in their multivariate predictive models. My advice is to concentrate on simple things like time to complete a design-make-test cycle (which it is not in anybody’s interest to exaggerate) and have faith in the metrics (which were revealed in the teachings of Pharma’s Finest Minds).

  3. anon says:

    Hype has existed all throughout human history. There will always be con artists who will try to sell a banana for $120,000, and there will be people who are willing to pay that price for one reason or another.

    It would make more sense to focus on general education and teach people how to value things based on facts, but nobody knows how to fix that problem.

    1. Isidore says:

      You are being unfair, it’s not just a banana but a banana taped to a wall:

      1. drsnowboard says:

        So it’s a docked banana in a good confromation? Take my money…

        1. drsnowboard says:

          confromation TM

      2. Art history says:

        Height of modernism was a urinal, signed by Duchamp.

        First search result in the website link.

  4. Unintended consequences says:

    “My own take is that one could start by targeting the worst excesses, and then work down to the borderline cases. The worst behavior, to me, is deliberate, conscious hype: doing it when you know that you’re coming close to the line or stepping over it.”

    This approach would make things worse not better. The problem with this philosophy is that it penalizes the moderate, the restrained, the knowledgable and the ethical. Asking people to self-police would filter out the folks who understand that there’s work left to do, and leaves signal only from the people who are callous self-promoters or who are honestly too ignorant to know the limitations of their p<0.05 cure for Alzheimer's. I think the filter has to be on the other side: teach journalists, the public, and investors skepticism.

  5. Nameless says:

    That paoer title is loaded with buzzwords. Maybe the authors are trying to overcome the noisyness of the publishing world?

  6. Eugene says:

    Should academic press offices have their releases reviewed and approved by the academics whose work they promoting? Or does it need independent review? A lot of the copy from Academic press offices seem to have a template that claim the work will have a dramatic affect on some real world problem (the current fashionable one is Climate Change).

    1. anon says:

      Academics often do approve press releases from their university press offices and it doesn’t seem to help.

      The problem seems to be a collective madness. Everyone knows the introductions and conclusions of many chemistry papers are meaningless — the authors, the reviewers, the editors, the readers — but nevertheless it persists. It isn’t clear how much harm it does, given that we’re all complicit and probably filter it out anyway.

  7. Dr. Manhattan says:

    “These are actual facts, based on my theories”. I forget where I read this, I think it was one of the article in the hilarious “Journal of Irreproducible Results”, but it always stuck in my mind.

    Sometimes it comes way too close to the truth to be comfortable. Looking at you, Holmes.

  8. dearieme says:

    What should be done if a whole field is corrupted?

    1. Underpriviledged classmember says:

      The proletariat should rise up and seize the means of production.

    2. Someone had to say it says:

      Move a way from amyloid beta and work on tau instead

  9. Magrinho says:

    For organic chemistry and medicinal chemistry – the situation is not perfect but it usually gets corrected over time. If an overhyped reaction doesn’t work then people stop using it.

    If PI reputation = stock price, then one might see big fluctuations over time but legacy is much harder to hype.

  10. Anon says:

    Unfortunately there will always be a market for fake news. National Enquirer, anyone?

  11. Geoff Shiton says:

    Pat Walters has an excellent example of dissecting the recent hype around AI

  12. Steve says:

    It is often not deliberate hype. Many academics seemed truly convinced that their project will deliver miracles in solarcells/cancer/batteries/Alzheimers/gas storage etc. When you look closely it is always too expensive, slow to make, impossible to scale up, only works in the absence of water, tested on dubious mouse models etc. But they dream…

    1. anon says:

      It’s not their problem to solve. Proof-of-concept matters the most to academics.

      1. zero says:

        That doesn’t explain university spinoffs that rake in cash hand over fist or the academics that run them…

      2. HFM says:

        When an industry scientist tells you how long something will take, double it. When an academic scientist does that, double the number and increment the units.

        As you say, academia only cares about proof of concept. Everything that happens after that is trivial work for lesser minds. No matter that our prototype is held together by chewing gum and grad student tears; we’ve done everything hard, now it should be on the market in six months. Unfortunately, the real world isn’t that simple…as the “lesser minds” who licensed the darn thing will spend the next 12 years finding out.

  13. milkshake says:

    here is hypothetical example: lets say you worked for a startup company that went all the way on hype-hypocrisy-falsification-fakery route, it was done initially to raise money and later to fool the investors and the potential buyers.
    There was lost of shady shit besides it that was covered up. But the research misconduct is already manifest in the published articles if you know where to look. How does one go public about it without destroying his own employability, and without getting tied in libel and breach of confidentiality agreement lawsuits for the next 10 years? Especially if you talked to the main investors from an capital investment group who poured tens of millions USD of their own investors money (that they can never get back), and they don’t want to pursue the culprits because they would only embarrass themselves in the process, and would look like imbeciles who ended up fleeced because they failed to do a due diligence?
    Let say you also spoke to people at NIH, whose million USD grant was used to fund an falsified research, and they do not want to be bothered by a messy research integrity case?

    1. DrOcto says:

      This does not sound hypothetical…

    2. crni says:

      Get a good lawyer, assess how much federal money is in it and see if you can retire on the whistleblower reward. No future employability needed then.

    3. dearieme says:

      Tell all and then spend years in either Moscow or the Ecuadorian Embassy in London.

  14. NotHF says:

    There are two problems: 1) the hype blowhards often have enough clout to, I don’t know, call you a terrorist at an ACS meeting when you question their trash science and still get invited to give talks places (link in the website space). 2) A lot of us who’d like to call BS on what we know are solutions looking for problems are early career and have a pretty legit fear of having it hurt our careers.

  15. Anonymous says:

    Anyone else catch this quote on page 2 not-so-subtly dissing Angewandte, the journal their essay is published in?

    “Another gimmick has been the borrowing by some science journals of the device of “quickie” personal interviews,traditionally found in the popular press, featuring answers from well-known scientists to inform the reader about irrelevant trivia such as their favourite food, drink or fiction writer.”

  16. metaphysician says:

    So, I tend to think this, like with many things, is at least partly a problem of short term thinking and incentives in corporate leadership. Thus, the solution I often propose: executive bonuses should have a time delay. Instead of getting bonuses as cash, or as stock options, the bonuses take the form of a stock trust, which only vests in the recipient after ( say ) ten years. Until that time, they can collect the dividends off of the account, but they *cannot* sell the stock. Thus, they have a strong incentive to not just have the value of the company be high now, but to have the value be high *over the following ten years*, because if the stock price collapses, so too does their account.

    Net result: strong incentive to stay with a company rather than jumping around every few years, strong disincentive to leaving a time bomb at your company, and coincidentally an incentive to actually pay out dividends ( which is what all stock *should* be doing, anyway ).

    Aside from the obvious ( none of the current participants in the system would want this, since it means they can’t pump-and-dump their own careers anymore ), any flaws shine out?

  17. SilicaLungs says:

    From the paper:
    “One has been the increasing pressure on academic scientists to demonstrate the “quality”, “value” and “impact” of their work, not through peer-conducted, in-depth evaluations of the extent to which their work is intrinsically important in uncovering new scientific knowledge, but rather through metrics”

    As if peer-review itself is not open to gaming strategies. Instead of trying to create a dichotomy for their arguments sake, the authors should have expanded upon how the peer review process could be reformed and made more transparent.

Comments are closed.