Skip to Content

The Dark Side

Dupeless Needication

Here are two papers have been going around on Twitter for a few days now. The first one is from a Hindawi title, “The Scientific World Journal”, from a group at the University of Malaya. And the second is from the same team (several overlapping co-authors), published a year or so later in Scientific Reports. Neither paper is, frankly, of very much interest as far as I’m concerned – you could probably publish these “Hey, this random compound does something to cells” papers every week if you wanted to. Every time I see these things, I can hear what Samuel Johnson said about “Ossian“, that is “A man might write such stuff forever, if he would abandon his mind to it.” But subject matter aside, the immediate problem is that Figure 4 from the first paper is the same batch of pictures as Figure 2 from the second one (well, slightly dimmer), and they’re supposed to be looking at completely different compounds. This isn’t possible, of course.

The University of Malaya took pretty swift action the last time this happened, and I expect that they’ll want to have a look at this situation, too. In a larger sense, though, what this makes me wonder is whether anyone has written image-comparison software to catch things like this automatically. You could start with an algorithm that calls up the papers from all the co-authors for the last few years and pulls the figures and images from each one, then starts sorting through them for similarities. I certainly have never programmed something like this, but it seems like you could pick some distinctive contrasty feature from a given frame and use that as a fingerprint to look through the others. If you wanted to get fancy (and people do get fancy like this), then you’d also want to have it search through some of the rotations as well. For all I know, there is such software already, but (not being a journal editor) I’ve had no occasion to seek it out.

If it does exist, though, it doesn’t appear as if many journal editors themselves have had occasion to seek it out, either. This sort of thing happens way too often. You have your duplicated gel bands and lanes, your duplicated cell pictures, and (a favorite) your cut-and-paste jobs that copy out individual features inside what’s supposed to be a single frame. Shady cell and molecular biologists do plenty of this, but you can find it in chemistry, too, of course: try this one and this one, both spotted by F. X. Coudert on Twitter. I mean, we already have enough problems with results that are hard to reproduce – does it help anyone to go on and fill the literature with actual bullshit? Just made-up stuff? This is what I think of every time I read about machine-learning programs that will whiz through the scientific literature and distilling out all that knowledge and all those connections – they’re going to be abstracting out this kind of stuff, too. Just today, Retraction Watch has word of nearly 60 papers being pulled from a bunch of Iranian “researchers” who were manipulating the review processes at Springer and BioMed Central to publish piles of plagiarized “results”. So before we start gathering all human scientific effort together, maybe we should make a couple of passes to remove all the crap.

54 comments on “Dupeless Needication”

  1. Anon says:

    I’m looking forward to seeing how IBM’s Watson can produce amazing insights by churning through all this BS. The only difference between the crap Watson feeds on and the crap it vomits out, is that more people will believe what comes out. With Big Data and AI, you can polish a turd!

    1. MTK says:

      I’ve wondered about this issue a lot. The amount of irreducible data in the biomedical field is, as we all know, immense. How can AI deal with this issue? Crap in, crap out.

      You would have to instill some sort of weighting to datasets that would place greater weight on those that have either been replicated, confirmed, or supported by others. Or place greater weight on studies from labs and groups that have had other studies replicated.

      It’s a thorny issue to be sure.

    2. Samira Peri says:

      Actually, Mythbusters proved that all you need is a suitable turd and elbow grease. No need for computrons.

      1. Phil says:

        If I remember correctly, lion turds were especially suitable for polishing.

    3. m says:

      I’m in no way a fan of “Big Data” let alone “Big Data” hype, but the whole point of something like Watson is that it is supposed to be able to ignore a nonsense paper like this, which shares a lot of similarities with other papers no one cares about. It’s old fashioned manual curation and indexing that would make this show up in a search result and not tell you its nonsense.

      1. Anon says:

        “the whole point of something like Watson is that it is supposed to be able to ignore a nonsense paper like this.”

        Good luck with that!

        AI/Watson can only look for patterns, and that would be strengthened by two different compounds showing the same result. But it will never have the ability to determine which result is true or false, because the information about absolute truth is simply not there. Truth can only be assumed based on what is seen.

        Basically, garbage in, garbage out. There is no way of beating entropy.

        1. m says:

          Getting the same result for a pair of unrelated compounds is a pattern, and one that in principle can be recognized and used to dismiss a paper. The idea that patterns always strengthen outputs, as opposed to downweight them, is as simplistic as “garbage in, garbage out.”

          I’m not even sure about your second point. Of course no AI is going to read papers that are fraudulent and come up with valid info based on the fraud? Is that all you’re saying here?

          If so we’re talking past each other. Even for all their hype, no one at IBM pretending you can feed Watson a fraudulent paper and it can draw truthful insights based on that paper. They are imagining you can feed in good literature and bad literature and it will know to ignore a lot of the bad literature. It is doing the ignoring, incidentally, based primarily on human conclusions about the value of the research, inferred through the topics of literature that come up more frequently. It’s not actually reading papers and judging them on it’s own–it’s a bit more than glorified citation counting, but that’s the basic idea.

          1. Anon says:

            “Getting the same result for a pair of unrelated compounds is a pattern, and one that in principle can be recognized and used to dismiss a paper. The idea that patterns always strengthen outputs, as opposed to downweight them, is as simplistic as “garbage in, garbage out.””

            So, how do you know when a pattern means that less weight rather than more weight should be placed on a result? And how do you know *which* result to dismiss and which result to keep (if any) even when you do want to dismiss one?

            The fact is, you don’t. The information is just not there, and that’s what “garbage in, garbage out” means. You can’t add information that isn’t there unless you actually do another experiment to get more and better data.

      2. Anon says:

        P.S. Can *you* say which of the two compounds corresponds to the dupicated figure, even now that you know they cannot both be true? Can you say for certain that any published result is true?

        No. And neither can Watson.

  2. Kevin Sours says:

    I don’t know of any specific app, but the basic technology for it exists:
    https://support.google.com/websearch/answer/1325808?hl=en

    It may be that you’d need to calibrate what you were looking for because to an untrained eye chemistry slides all look the same. But given the kind of image matching people are already doing it’s a matter of doing it rather than figuring out how it could be done.

    Somebody should actually chat with Google about it, because applying their technology to random projects that benefit society is something they do on occasion.
    Kevin

    1. loupgarous says:

      As long as they do it better than Google Scholar, which several scientific bullshit artists have gamed so that otherwise intelligent people look at their GS h-index and conclude they have a positive impact on their fields when all the h-index does is count how often a given scholar’s been cited. It doesn’t care who cited him (“himself” seems to be plenty) or in which journals.

      This has lead to a gold rush – in “gold-model” open access journals which support their operations by charging manuscript preparation fees and hefty reprint charges – all very much worth it to fringe scientists and outright academic frauds who crave recognition (or in one case has run at least two apparent “pump-and-dump” penny stock scams based on his “discoveries”).

      Depending on Google to help muck out the Augean stables of academic publication fraud is leaning on a slender reed. Google Scholar made the problem worse.

  3. Anon says:

    “this makes me wonder is whether anyone has written image-comparison software to catch things like this automatically. You could start with an algorithm that …”

    … checks whether the work and/or authors come from Asia?

    1. loupgarous says:

      ,,, or Africa, or Europe, or Florida, where one guy’s built a business empire based on bogus discoveries documented in “gold model” open access scientific publications.

  4. IGM says:

    For those who don’t know about it, I highly recommend the seems-like-magic image-comparison site tineye.com. I don’t know how they do it, but it works. I just wish they had more images in their dataset.

    PS The captcha for this was “six + blank = ten” but you are not allowed to type in “four”, oh no, it has to be “4”.

  5. The problem is an important one and currently gets addressed by researchers surreptitiously. Let’s assume for a second that finding image dupes can be automated. While there’s a substantial return on investment (i.e. avoiding experiments or lines of research).
    Who wants to for development, operation and maintenance of such a system? Let me know – we may be interested.

  6. Just tested if google image search or tinyeye find the dupes. They don’t. (fig 2 in http://www.nature.com/articles/srep09097 and fig 4 in https://www.hindawi.com/journals/tswj/2014/540463/fig4/) are

  7. road says:

    I’m guessing the hardest part would be getting access to a database of journal figures. An entity like Elsevier could certainly do these sorts of analyses *within* their titles, but I can’t think of any easy way to search across titles and publishers. The image-comparisons are the (relatively) easy-part. Perhaps there’s a way to access figures from titles in Pubmed Central, though…

    1. loupgarous says:

      Assuming Elsevier cares – and they’ve been wildly credulous in publishing crap (such as the existence of “Magnecules” of hydrogen no one but their purported discoverer and his posse have published seeing – certainly, no independent body of researchers who’ve made their raw data and an exact tally of their methods and materials accessible to those who’d like to replicate the study findings).

      So getting Elsevier to police figures and graphics for drug discovery research is a lost cause. If you’ve got the money, honey (apologies to Hank Williams, Sr.), they’ve got the journal space.

  8. Derek Jones says:

    Defeating image recognition software is a cat and mouse game. It did not take long for somebody to figure out how to defeat recursive neural network classifiers (the latest in hi-tech techniques): https://arxiv.org/abs/1610.08401

    1. Confusedius says:

      Er….what?

    2. loupgarous says:

      Thanks for the cite! And while the overlaid image that confounded these prominently-used image recognition algorithms was easy to see against a light background (such as the sky above the killer whale misclassified by the universal perturbation as a “African grey”), it wasn’t so easy to spot over darker and more uniform frames, pointing to additional difficulty in subtracting universal perturbations from original images so image recognition software isn’t confounded by them.

      Second moral to that story: GoogLeNet is among the image recognition architectures the authors of that paper were able to fool about half the time regardless of which architecture the perturbation was optimized to fool, and 78 percent of the time when the perturbation was optimized to fool GoogLeNet (by drawing from the image pool used by GoogLeNet to compute the perturbation).

      But having someone use advanced, experimental image computation software to create a frame full of faint wiggly lines to fool image recognition software is lowering the water when it’s easier just to raise the bridge by not using identical images to document scientific findings for different experimental compounds (or, in the case of the University of Malaya team Derek busted in this blog, the “before” and “after” images for the test of the same compound in the same paper). That team was guilty of, as Derek aptly put it, “deliriously incompetent fraud”. The team should have hired Theranos to do their presentation.

  9. Paul Brookes says:

    Some time ago Konrad Kording’s lab in Chicago (http://kordinglab.com/) was working on exactly this. However, seeing the disparaging comment on “machine learning” in Derek’s post makes me hesitant, because that’s exactly the approach they were using… train the algorithm using a databse of similar looking images so it can detect similarities in an unknown set.

    However, the real problem is not the lack of such tools. Rather, it’s that even when they exist the publishers will not spend their precious $$ to use them. The fact that such software is often free, but we still keep hearing about new plagiarism cases in 2016, is a clear indicator of precisely how many ****s the publishers give about this problem.

  10. Haftime says:

    As road pointed out, the reason why this doesn’t exist isn’t a technical issue, it’s a social issue. Journals have extremely restrictive scraping rules, even for academics (they vary with publisher widely). Peter Murray Rust at Cambridge has been trying to automate collection of this kind of information with projects like The Content Mine ( http://contentmine.org/ ), but has had great difficulty getting the access to the data ( https://blogs.ch.cam.ac.uk/pmr/).

    A related example where this works would be from crystallography, where raw data is routinely deposited and checked – the Acta Crystallographica E scandal – where the equivalent kind of fakery with crystal structures (a series of fake structures made by swapping a transition metal atom) was uncovered by automated checks on the raw data.

  11. Li Zhi says:

    Yes! We need to off-load more of our joint responsibilities for maintaining the “commons”. Is peer review a commons, or is it each discipline’s literature? Anyway, the publishers (i.e. “somebody else”) need to be made responsible and should pay for it. Seems to me many publications already make figures electronically available without cost, maybe the big Societies need to address this. Should there be some sort of standards for access? I’m dubious that an A.I. will be able to do this anytime in the near future. Its like making something foolproof; the fools are far too clever. Take facial recognition software, my understanding is despite the enormous amount of time and talent invested, a (fake) tattoo, some mascara and sunglasses are more than sufficient to defeat it.

  12. siliam says:

    Given it’s other abilities to detect transformations, I think google could set up TinEye to do this in a bored afternoon, given access to the materials…

  13. loupgarous says:

    Google Scholar doesn’t help the matter. They advertise their “h-index” as an indicator of an author’s influence in his field by counting number of citations, but the people who are over the academic fringe all the way into academic fraud and academic imbecility have found how to game it by “log-rolling,” citing each others’ essentially useless papers published in “gold model” open access journals. That’s how the eminent Ruggero Santilli who thinks Steven Weinberg and other Jewish scientists prevented him from getting published in peer-reviewed physics journals, and who sells “Santilli telescopes” with concave lenses that see antimatter galaxies and “Invisible Terrestrial Entities” has an h-index higher than many reputable researchers who’ve made genuine contributions to their fields.

    Google Scholar is doing the exact opposite of what I hope they intended, giving a quick and easy way to gauge the impact of a scholar in his field. We’ll just have to go back to reading the papers and deciding for ourselves. Counting citations won’t get the job done.

  14. Anon says:

    One could invest millions of dollars to increase the probability of detection 10-fold, from 0.1% to 1%, or simply increase the penalty from a temporary publishing ban to execution at no cost.

    Same impact, but a lot cheaper.

  15. Dave Fernig says:

    There remains the human being to do the job – if it’s Ossian, I don’t read. Or cite. Or go to a meeting where the purveyors of bull parade in front of 100s if not 1000s.
    Another good filter is hype – the greater the ‘miracle’ the less chance there is anything other than misconduct underneath.
    Finally, PubPeer is most useful, since other than just my own eyes, there are 100s if not 1000s of pairs of eyes reading and providing me with their insight.
    Cleaning the literature is not possible with the present publishing and reward model; an open data publishing model would change a lot, since original data files are in easy machine readable format.

  16. pharmasteve says:

    People are quick to criticise this type of issue with modern chemistry literature, but there are more and more unquestioned made-up yields and selectivities coming out of “high profile” labs that blind eyes are turned to. I challenge anyone to reproduce any Gaunt methodology and/or notice that there are claimed reactions that are often nowhere in supporting informations.

  17. Peter Kenny says:

    The article discussed in the blog post linked as the URL for this comment may be of interest?

  18. Ali Gherami says:

    In my personal point of view, authors of these 2 papers have published the second paper as a copy paste job only. They 100% knew what they are doing and they can’t even evade that they were not aware of this duplication. The paper discusses based on these fake figures and expands through the paper so all of the relevant discussion also can be wrong because authors the scaffold of this paper is fake data. The role of BrdU and Phospho-Histone H3 which broadly discussed in the paper and is relevant to further analysis is a 100% fake data. The bar chars are also incorrect. The flowcytometry related ro the experiment could be also invalid because they suppose to support each other so once one is faked, another one is not supportive.
    I feel University of Malaya should rectify their education and publication system rather than publishing 3823 papers which most of them are low in data quality and some are absolutely fake.

  19. There is no incentive for publishers to police and correct their literature. Why shoot themselves in the foot? They prefer to let things drag out for months or even years, all the while pumping out volumes more erroneous or problematic literature than that which is corrected. I’m afraid vigilantism has reached science’s main street and with or without software, we are all responsible for cleaning up this mess we are all in.

    1. SedatedFMS says:

      It’s the Daily Fail…………..

    2. Derek Lowe says:

      Johnson was on to Ossian’s real origins from the start – when he was asked if any man living could have written the poems, he said something to the effect of “Why, yes, many men. And many women. And many children”

  20. Ali Gherami says:

    Now this question will arise that which compound and which result is true?? I guess both have serious problems. we can not say really the same duplicated figures are related to the Copper based compound or they are related to the Schiff based in Scr.Rep. In fact which one has done first and duplicated later!!!??
    A big confusion and question mark is present here. Of course, Sci.Rep is a copy paste job but is the integrity of Hindawi also true? are the faked figures related to Hindawi indeed?

    1. Anon says:

      That’s the key issue here. When you add a fake paper, not only do you add zero value, but you damage the credibility of papers that are true. The cost of additional uncertainty and entropy is huge: If 50% of published papers are false, then 100% of papers are worthless because it becomes a random coin flip. May as well flip an actual coin without reading any papers. And that’s pretty much where we are right now.

      1. Ali Gherami says:

        Yes absolutely right. Once a misconduct occurs, it will effect not only one case but in number of cases indeed. This misconduct has effected those authors who have cited these 2 invalid papers, however the number of citations is high. Lots of people will be going to be in trouble now. Science integrity is what Malaysia not following but they just publish their low quality data in open access journals. Just have an overview of their publications and you will find more than 75% are published in open access journals which are X100 times easier for authors to publish their data rather than non-open access journals.

    2. Anon says:

      PS. As Mr. White said in the movie Spectre:

      “Money isn’t as valuable to our organization as knowing who to trust.”

      The same could be said of the scientific community.

      1. Anon says:

        Sorry, that was actually Casino Royale. 🙂

  21. JK says:

    Automated detection sounds nice, whether the problem is finding a technical algorithm or overcoming paywall archive access.

    Unfortunately it does not really get to the heart of the problem. Derek asks “does it help anyone to go on and fill the literature with actual bullshit?” At present the answer appears to be yes, namely the authors of said BS. If we automate the detection of duplicate images then people will find another way to cheat – it’s really not that hard. Duplicate images have come to attention because we have an easy way to spot them. But is there any reason to trust any number in a paper with faked images? Can we trust any number in any paper by the same authors? Changing a number is even easier than changing an image. The obvious suspicion is that undetected fake numbers are out there at even higher rates than fake images.

    I don’t have any easy answers, but anyone who cares about scientific knowledge for whatever reason needs to be very worried. We need a culture in which honesty is rewarded above all else and in which the idea of faking data is all but inconceivable. Finding ways to value results that reproduce must surely be part of the answer. I guess the citation cult got started with just this idea in mind, but it hasn’t really worked out. Without some solutions it is hard to see how science can survive.

    1. Ali Gherami says:

      I am on supportive of your idea. These authors are not trustable any more. They have published lots of papers based on the same theory of these papers. Only have chosen different compound, but the methodology is same. It is not anymore interesting for science to repeat a work more and more. I haven’t gone through all of their papers published so far, but I am sure I can find more falsified data in other papers from the same group too. Just look at their names in other publications only. The authorship is only shifted in most of the publications but names are same. A fake group with a plan maybe !!!!

    2. loupgarous says:

      I don’t think any automation exists that can counter good old-fashioned academic gamesmanship when it comes to experiments unlikely to be confirmed by following a paper’s “methods and materials” section. The faker would just have to exercise the same care millions of “but indifferent honest” recent college graduates take with their resumes, and hope no one does due diligence.

      In that regard, science is in one of its regular fits of growing pains, with even the physical sciences vulnerable to the same sort of “Better, Faster, More Comprehensive Manure Distribution” the social sciences have been afflicted with for over two centuries.

      For a while physical science had policed its own formal scientific publications by the practice of replicating extraordinary or seemingly implausible results of experiments, but even then there were horror stories. Read John D. Clark’s Ignition! for vignettes both of such implausible results, and real results which cost the people publishing them because (in the case of the discovery of chlorine pentafluoride, people who had the power to quash research punished the researcher without duplicating the experiment described to confirm or deny the results.

      Due diligence is what is needed, and what has kept science honest – the scientific method relies on independent confirmation of experimental data. But intellectual laziness leads to most cases of scientific dishonesty, and allows it to flourish unchallenged.

      I think the answer is another use of information technology – the establishment of rings of researchers in the same specialty fields who don’t just peer review results in journals, but can be relied on to replicate the experiment while giving priority to the researcher who submits his findings for “experimental peer review.” If this is done prior to publication, it could be a good way to strengthen trust in the reliability of scientific papers from new researchers who claim exceptional results (or are trying to build a strong record of less momentous publication). It would also expose frauds much more quickly – these people would avoid submitting to a specialty research ring or be caught by workers who know the problem the submitter is writing about and can rapidly run the experiment themselves to affirm its value or find issues with it.

  22. trip says:

    How can a computer ever be smart enough to know that adding thiolphenol to already published reactions IS NOT plagiarism but rather a major advance?

  23. Ali Gherami says:

    The solo matter that can be done by any author is honesty in publication. If authors are honest to science, their work is always appreciated but if they can’t, better to leave the science and do something else.

  24. MTK says:

    The title of this post is funny because it’s not dupeless at all.

    The authors tried to dupe the scientific community.

  25. Boris Barbour (PubPeer) says:

    Such detection software exists

    http://www.nature.com/news/image-search-triggers-italian-police-probe-1.14295

    and there are rumours that journals now quite often try to detect duplications. However, it’s only a partial solution to the problem unless robust public action is taken, such as disciplining researchers and alerting the community. This rarely happens.

    Note that sites like PubPeer can centralise such information when it is discovered, so that repeat offenders are identified to the community.

  26. Ali Gherami says:

    Why Scientific Reports doesn’t place an expression of concern for this fake paper? damn hilarious.

  27. Anonymous says:

    Ali Gherami claims on Twitter that he was banned from commenting on PubPeer. Can Ali and PubPeer please clarify the veracity of this claim.

    1. Ali Gherami says:

      Yes, I confirm that I was blocked by pubPeer due to arising the concerns about these 2 fake papers. I still need a complete answer from Pubpeer? Is posting a fake paper wrong?

    2. Ali Gherami says:

      Check my Twitter (@aligherami1), the post is still there.. Shameful behavior of PubPeer.

  28. Scott says:

    The US military has had a *very* good image-comparison system running for years, it was used pre-GPS in Tomahawk missiles. It’s called Digital Scene Mapping Area Correlator, or DSMAC. https://en.wikipedia.org/wiki/DSMAC#DSMAC

    Bet it’d work here.

Comments are closed.