Skip to main content

The Scientific Literature

Results From the JACS Challenge

The first results are in for the “JACS Challenge” that I mentioned here the other day. It looks like people’s picks did correlate with later citation counts, overall, but with some curious exceptions. Some things that looked as if they would have been referred to a lot weren’t, such as a paper on solventless polymerization. And a paper on transition-metal catalyzed boron-nitrogen bond formation was only picked by two respondents, but has amassed 258 citations since it was published. For the most part, though, people tended to judge the past pretty well (here’s the PDF with all the statistics, if you’re interested).

5 comments on “Results From the JACS Challenge”

  1. fuelcelldave says:

    I was one of the two to choose the transition-metal catalysed boron-nitrogen paper and might have selected it twice (didn’t write down my answers). For me this shows how subject specific this test is since I thought it was significant but with a background in main group chemistry I was in a position to know. Some of the papers I considered outside my specialities were way down the list.

  2. Anonymous says:

    Correction not one of the two since not in the first group.

  3. CET says:

    Is there a graph somewhere of citations vs. responses? (I don’t see one, and I don’t have time to fight with excel’s import function before my next class)
    Just scanning the tabulated numbers . . . I don’t know how predictive responses really are. The list sorted by responses seems to show a pretty limited correlation.

  4. Deep Lurker says:

    I’m getting a warning that the sciencegeist site is infected with a trojan.

  5. eugene says:

    I didn’t participate in the challenge, but I for sure thought the boron-nitrogen paper would be ‘significant’ because I have some background in the area. And that’s why I didn’t participate after I saw it. Because five years after it was published it was still a big thing. But ten years? It was all a fad of hydrogen storage and that’s why it got cited so much. But all that simple chemistry was never going to overcome the practical challenges of using these materials in actual real-life situations. Certainly that’s why industry never adopted on it; they have people who think about chemistry in practical terms. But the academic papers were certainly sold as practical and as showing a lot of promise for industrial applications, but it was definitely not true. It’s just that a lot of labs saw a way to publish quickly and get a lot of citations, then everyone forgot about it. It’s dead now.
    To me it’s a classical example of wasted effort and misuse of resources. Any logical look at the system will tell you it’s not suitable and you should explore other strategies if you want a hydrogen storage system. But I guess if you want Jackass glory quickly and a student needs to finish up a crappy project, then you keep working on it. I even met one of the PIs working in the area at a conference back when the field was hot five years ago, and he said to me (after a few beers) that this is all crap and never going to go anywhere and he was trying to wrap up and get out of the area.
    The >200 citations are a testament to how the impact factor system is a failure in judging true significance.

Comments are closed.