Skip to Content

General Scientific News

Large Teams and Small Ones in Science

I had a book review recently in Nature, on a new volume (Thrifty Science) that looks over the history of early scientific experimentation from the viewpoint of its frugal nature – the idea of reusing and repurposing equipment, objects, and even rooms in one’s house. There was indeed a lot of this sort of thing, as the book makes clear, but I wondered about one of its conclusions – which was that this sort of thing could well be making a comeback. I’m not so sure about that. While there are things that can still be discovered by basement experimenters and make-do apparatus, my own belief is that the bar to discovery has been creeping inexorably higher just because of the nature of science. We build on each other’s work, and we’ve built on an awful lot of stuff that was discovered with simple equipment and under simple conditions. So over time, working in any of those areas tends to become progressively less simple.

A countervailing trend is the availability of such equipment, but the stuff that’s widely available tends to be so because lots of people have already done lots of research with it. If you’re going to make new scientific discoveries on your own out in the garage, you can get plenty of apparatus, but you’re going to have to be more ingenious than ever to come across something that no one has has already explored. Robert Boyle and the rest were indeed great scientists (Newton deserves his high reputation even after any historical adjustments you care to make), but they did have plenty of open room to run in as well. And the open spaces of today, it seems to me, are less and less available to thrifty home experimenters. Mind you, I wish that weren’t so. But I don’t think that the scanning tunneling microscope (to pick one example) could have been prototyped in a garage lab. One response to this is “Well, PCR could have been”, but think about the state of molecular biology when PCR was first developed – I’m not so sure about that at all.

This recent paper in Nature has perhaps some bearing on this topic. The authors are looking at a measure of the impact of published papers:

Here we analyse more than 65 million papers, patents and software products that span the period 1954–2014, and demonstrate that across this period smaller teams have tended to disrupt science and technology with new ideas and opportunities, whereas larger teams have tended to develop existing ones. Work from larger teams builds on more-recent and popular developments, and attention to their work comes immediately. By contrast, contributions by smaller teams search more deeply into the past, are viewed as disruptive to science and technology and succeed further into the future—if at all. 

They quantify this by an ingenious bibliographic technique. When a paper gets cited, do those citing it also cite many of the references in the original paper? If so, that original paper is more likely to represent work that’s consolidating or extending a field that was already somewhat worked out. By contrast, if a paper is cited, but more in a solo manner, it is more likely to represent a new direction by itself, leaving those coming after it to cite it alone without other examples of something similar. To go to extremes, at one end you have review articles – very useful, but deliberately not breaking any new ground whatsoever. And at the other, you have one-off reports (at least at first!) of something that no one’s ever thought of or tried.

Applying this “disruptiveness” metric works surprisingly well – for example, papers that are known, in retrospect, to have contributed directly to major discoveries (as measured by eventual prizes and recognized impact) do indeed rank higher on this lone-citation scale. And when you look at the number of authors on all these papers, you find a very noticeable trend: the number of such “echo citations” (my phrase) grows as the size of the team grows. Which means that average disruptiveness goes exactly the other way. When you look at the set of most-disruptive papers, they are much more likely to have been smaller-than-median teams, for any given scientific discipline. This distribution also holds for the patent databases, and even for code and routines published on GitHub. Differences in the subjects being researched and the way the experiments are (or have to be) set up are real, but the team-size factor is larger than any of them (although the relative sizes of the teams varies in each data set). If you look at the top 5% in each category (developmental versus disruptive), the graphs shown above diverge even more robustly.

Think, for example, of the famously huge author lists found in high-energy particle physics. If there are seven hundred and forty-three authors, the resulting paper is more likely to be the long-sought confirmation of some extremely difficult-to-observe phenomenon (like the Higgs boson) whereas the paper that proposed that something like the Higgs boson had to exist will have had nowhere near such an army behind it. Such ideas take a while to catch on to the point that you can get seven hundred coauthors to work on them – and such large teams are more vulnerable to failure if the whole idea is wrong, too.

But this team-size effect holds up from almost every angle. The relative disruptiveness of review articles is not large, for example, but the most disruptive ones have the smallest number of authors. If you remove self-citations or try cutting out all the references but the high-impact ones, the effect is still there. Controlling for publication year doesn’t get rid of it, nor breaking things down by scientific discipline. Controlling for authors themselves actually make the correlation more robust: any given scientist’s most disruptive papers are generally those published with the smallest number of co-authors. Looking at the patterns of the citations themselves, it appears that smaller teams (and solo authors) tend to reach back to older and/or less popular ideas, rather than chime in on something that’s already rolling along. It’s for sure that more of these small-team papers disappear without much of a trace, and they tend to have a long delay before citations pick up, but when they have an impact, it’s a larger one.

The lesson of this paper is not “Small teams good, large teams bad”, though. Both kinds of work are needed, and they’re part of the normal development of scientific ideas. I was gratified to find that the paper addresses the washing-your-car-to-make-it-rain problem: you don’t necessarily generate disruptive work just by artificially forming people into small teams. That’s not how it works – disruptive work instead tends to cause smaller teams to form around it. Funding small teams by traditional means (renewable grant proposals, for example) may well be a recipe for filtering out what could have made them interesting in the first place:

We analysed articles published from 2004 to 2014 that acknowledged financial support from several top government agencies around the world, and found that small teams with this funding are indistinguishable from large teams in their tendency to develop rather than disrupt their fields. . .This could result from a conservative review process, proposals designed to anticipate such a process or a planning effect whereby small teams lock themselves into large-team inertia by remaining accountable to a funded proposal.

To circle back to the topic I started off with, it would also be interesting to know what sort of facilities and equipment correlate with disruptive work. There probably is a tendency towards increasing cost and complication as you go to larger teams (it’s hard to see how there could not be). But it’s important to get that causality right: giving people fewer resources is probably not the recipe to make them more inventive, since the great majority of people will not respond in the manner you’re trying to induce. And my guess is that the relative technical sophistication (and cost) of even the smaller teams, for a given discipline, has increased over time as well.

 

 

20 comments on “Large Teams and Small Ones in Science”

  1. navarro says:

    “. . . giving people fewer resources is probably not the recipe to make them more inventive, since the great majority of people will not respond in the manner you’re trying to induce.” This observation has relevance to my field. I realize that comparing education and teaching to scientific discovery is somewhat of a grapefruits to peaches comparison, if not a tire rims to peaches comparison, but this statement does remind me of some things I’ve heard over the past 24 years. Indeed, I have heard superintendent level administrators say that “Money won’t solve the problems of education. The more money you give to teachers the less creative they are.” I’ve heard folks from the agency over education in the state in which I live and work say that giving districts and teachers more money to work with would only decrease the level of innovation our teachers display. I’m not sure how they’ve come to their conclusions but they seem to believe, or at least hope, that teachers will respond to adversity by doing more and being more creative.

    1. Hap says:

      When they say that, I’d like to pay them less and see how innovative they get.

      I think more money and larger teams tends to be a safety play – you want to make sure you don’t fail, not that you succeed really big. People that don’t have money (or smaller teams) are likely to either succeed big or fail, and probably much more often fail. If that’s true, then less money means you’ll get a population with a few big wins and many losses (either a few good results or a few good schools and educated people). It’s probably worse to have a feast and famine system rather than spending money for consistency – I don’t know if the overall benefit is positive or negative, but there will be lots of bad stuff from the people who fail and eventually feast/famine systems are likely to fail when the faminers go after the feasters or are aroused by an almost-feaster.

    2. aairfccha says:

      The problem with that thinking is that it considers creativity as a goal in itself rather than a means to an end. Even if less money is stimulating creativity to compensate for the money shortfall, results are still quite likely to lag behind what you would have gotten with proper funding.

    3. Anonymous says:

      navarro, “fewer resources … stimulate creativity” — I have seen writings on both sides (at least w/r/t science and scientific research). Many say that stress caused by a lack of resources, low pay, admin pressure to produce results from nothing can have an impact on creativity and productivity. Some say negative, some say positive, and they say it to varying degrees.

      But there are other generalizations that apply, such as (for chemist-Pipeliners) ‘no matter how much bench space you are allocated, you will always fill it up!’ More generally, you will often use all available resources to do your best on a project. If you have more resources, you will think of more ways to make it better and to do that ‘one more thing’ to strengthen or improve it. With fewer resources, you may wish for better support and you might even pull an HC Brown (see my other post on this topic) and figure out how to make do with what you’ve got, even if it takes longer or isn’t as accurate.

      There is another generalization, from surveys I have read about (it’s psychology, so there may be a lot of wiggle room or complete refutation), that no matter how content they are, people typically want more than what they already have. Even if they admit that they are getting by, low income earners typically say, “If I only had ~10% more, I’d be able to replace the ’69 Chevy with a ’92 Ford.” But high income earners say the same thing! “If I only had ~10% more, I could buy another vacation home, this time in Tahoe.”

      Having been in poorly resourced situations and in well funded situations, I can assure you that I prefer the latter and I do not feel that it suppresses my creativity.

      1. navarro says:

        You make some good points. From the teacher perspective, most of us have developed an attitude of “as long as you’re not slapping me in the face while I’m trying to eat, it’s okay.” Unfortunately some states, since each state is in control of its own pay scale etc., have decided to reduce teacher salaries, add unpaid days and hours to the school year, or both. That has led to some of the teacher strikes you’ve seen over the past year even in states unfriendly to unions and strikes like Oklahoma. I guess adversity really DID spur creativity.

  2. MrXYZ says:

    The story of Eric Betzig and his co-invention of the super-resolution microscopy technique PALM is an interesting example of what you are thinking about in your last paragraph (https://arstechnica.com/science/2015/04/quitting-failures-a-microscope-in-the-living-room-nobel-prize/). It was definitely shoe-string work (complete with living room microscopes) but Betzig, Hess, and all their collaborators were incredibly technically sophisticated.

  3. tlp says:

    I see here some general principle that ‘to thoroughly develop an idea requires much more material resources than to generate it; the latter in turn requires much more mental resources’, which also applies to the topic of academic vs industrial drug discovery as well as many others (e.g. science vs engineering). Sort of like Pareto principle with additional physical-vs-mental dimension.

    1. loupgarous says:

      tlp said on 20 February, 2019 at 12:55 pm:

      “I see here some general principle that ‘to thoroughly develop an idea requires much more material resources than to generate it; the latter in turn requires much more mental resources’, which also applies to the topic of academic vs industrial drug discovery as well as many others (e.g. science vs engineering).”

      It’s a useful principle in the field of drug development. It allows for sharp distinction between
      – the resources needed to discover the foundations on which new drug families are based (primarily academic research) and
      – the resources needed to thoroughly develop that foundational knowledge into useful new drugs and assure they are better therapy than existing drugs, and more beneficial to the patient than they are toxic.

      Smaller teams are involved in foundational research than in drug development (multi-center clinical trials can involve dozens of co-authors).

      The intellectual scope of drug development work is necessarily narrower than foundational (“what happens if we inhibit this protein’s function ?”) research, and its costs are much higher. The scope for drug development research to be disruptive is mostly confined to serendipitous findings in clinical trials such as the ability of PDE5 inhibitors to treat erectile dysfunction (they were being studied as cardiology drugs at the time).

  4. AQR says:

    “Submarine” or “Blue Sky” research is a case in point. The work is by its very nature speculative, maybe even a bit off the wall, and can only be pursued with minimal resourcing. Nevertheless, it is seen by the researcher as sufficiently valuable that they are willing to devote some time and resources to it rather than on the work for which they will be officially evaluated. Of course, the large majority these efforts never bear fruit, but the ones that do hold the potential of being disruptive. The inventor/author will be that one researcher with the insight who was willing to take the chance.

  5. loupgarous says:

    This recent paper in Nature has perhaps some bearing on this topic. The authors are looking at a measure of the impact of published papers: (… ) When a paper gets cited, do those citing it also cite many of the references in the original paper? If so, that original paper is more likely to represent work that’s consolidating or extending a field that was already somewhat worked out. By contrast, if a paper is cited, but more in a solo manner, it is more likely to represent a new direction by itself, leaving those coming after it to cite it alone without other examples of something similar. To go to extremes, at one end you have review articles – very useful, but deliberately not breaking any new ground whatsoever. And at the other, you have one-off reports (at least at first!) of something that no one’s ever thought of or tried.

    The relative ease with which one can read supporting citations in papers one cites to establish a fact in a modern research paper may skew the metric the authors of that paper in Nature are using.

    I was a medical writer in the old days before medicine made wide use of the Internet to publish papers. My employers, a growing cardiology center with clinics throughout the Louisiana Gulf Coast, were careful men and women of science who examined useful facts in their intellectual context when publishing reports on their own research. This required periodic trips from the facility where I worked to the university medical libraries of New Orleans to make photocopies of every journal article we wished to cite, and moreover, copies of some of the articles cited by that article.

    Nowadays, I can do all of that without stirring from my chair, if I had access to every article of interest which is behind a paywall. The authors who identify this “disruptiveness” metric should examine the effect the Internet has had on making it remarkably more easy to go deep in citing supporting citations and the articles they, in turn, cite.

    It’s easier to cite other people’s research in what used to be considered great detail, now. Google Scholar’s publication of “impact metrics” for authors of published research has created an odd sort of log-rolling between authors who wish to appear to have more impact on their fields than they actually do. “Predatory” pay-to-publish journals depend on people paying to publish papers which either have marginal impact on their fields or are entirely fraudulent – but are cited by the authors’ friends, in exchange for having cited them.

    One hopes that this study of “65 million papers, patents and software products that span the period 1954–2014” winnowed out the garbage papers that even reputable pay-for-publication and open access journals have had to retract in that time period for academic misconduct and articles which never saw meaningful peer review, but were published because their authors’ check cleared the bank.

  6. Anonymous says:

    Nature paper: Paywall!!!

    In some of his autobiographical accounts, H. C. Brown (Nobel Prize; boron chemistry) describes the limited resources at his first independent position at Wayne University (before it was renamed WSU). He had very little equipment so he had to come up with research and experiments that could be done with what he had. One of those experiments (I think, … this is all from memory now), was to measure some physical properties, such as vapor pressure, of some compounds. That he did with some hand drawn capillary tubing and a lousy vacuum pump. He goes on to tell how, years later, some guys at MIT, using very fancy and very expensive equipment, verified his measurements (got the same values) which had cost just a few dollars for the glass tubing and a salvaged vacuum pump. “From little acorns, mighty oaks grow.”

    Although he did beta-lactam synthesis as a grad student with Sheehan, Corey started his independent career doing more physical organic chem than natural products synthesis. I wonder why? Maybe it’s easier to publish a result, no matter what it is, from P Org Chem whereas nat prod synthesis is riskier: nobody really wants to publish a failed route(s).

    “… STM … in a garage lab.” — This comment is about the Westinghouse / Intel / Regeneron Science Contests (for pre-college students). One winner, several years back, was a student who built a high energy laser in her garage and did some experiments with it. It kind of helped that her father was a PhD physicist at IBM or Intel or similar and provided guidance and access to professional labs, equipment, and numerous helpful colleagues where she was able to learn about what she was supposed to be doing at home. I estimate at least $100k worth of research funding (not counting the no-cost consulting) to build the laser at home and do the other research. Another winner was a student in the Boston area who did his research over the course of ca two years at his father’s biotech company. Likewise, probably at least $100k of “free” research funding for him. When I was in high school, using “advanced” books from the local low-tech public library, I set out to build to a spectrometer using cardboard tubes, Al foil, … and a $2 prism from the local hobby shop. (I think I still have the prism!) If I had been able to get into a lab to see a REAL spectrometer, even just for 5 minutes, I think I would have been able to spend my time (and $10 budget) more productively … on pizza. (I don’t see how “have nots” have any chance to win such science contests with such a lack of access to better resources. I’m talking about LAB sciences, not math or co sci.)

    AQR, “many never bear fruit” — Disruptive or a nothing-burger? Starlite en.wikipedia.org/ wiki/ Starlite was a one-man one-kitchen chemistry discovery that got a lot of press in the 1970s-80s and the attention of NASA and other large national and private labs. It seems to have fizzled out. Maybe it was too much Starlite that kept it from catching on fire.

    Betzig example: He still had access to expensive equipment and resources.

    Rules, regs, Homeland Security, etc. — It has come up many times In The Pipeline that these are no longer the days of Max Gergel’s Columbia Organic Chemicals or Elmer Fike’s Fike Chemicals. You can math, computers, even some biology in a kitchen or garage lab, but not chemistry. Just placing an order for some chemicals or equipment gets you on a watch list and possession can get you a prison sentence.

    Also to mention, if it isn’t mentioned in the cited works (book, Nature paper), is Derek J. de Solla Price’s “Little Science, Big Science.” It is a broader picture of the growth of science in general and the growth of Big Science itself.

    1. NJBiologist says:

      “One winner, several years back, was a student who built a high energy laser in her garage and did some experiments with it. It kind of helped that her father was a PhD physicist at IBM or Intel or similar and provided guidance and access to professional labs, equipment, and numerous helpful colleagues where she was able to learn about what she was supposed to be doing at home. I estimate at least $100k worth of research funding (not counting the no-cost consulting) to build the laser at home and do the other research.”

      As a judge for a high school science fair, this kind of thing frustrates the living hell out of me. Every year, there are kids who obviously had access to well equipped labs and substantial “consulting” support. Many of them ran with it, and more power to them. But every year, there are also kids who struggle to get access to equipment, reagents, etc.; I’ve seen a presentation that was basically supposed to be LC analysis, but the school’s LC packed it in and they didn’t have the budget to fix it, so the bulk of the presentation was the kid’s attempt to homebrew a system.

    2. dave w says:

      Re: regulations etc. – in many cases it’s the vendors themselves who have a “we only sell to Companies, not Individuals” policy – for example: http://www.espimetals.com/index.php/faq
      ” … we are restricted from selling to individuals. Because we sell materials which are hazardous we must verify that our sales are made to established companies. Before we can sell to any company, even if the purchase will be made using a credit card, we must receive and verify a bank reference and three relevant trade references.”

  7. Chris Phoenix says:

    In his book _Discovering: Inventing and Solving Problems at the Frontiers of Science_, Robert Root-Bernstein develops the observation that researchers tend to be innovative for their first ten years in a new field. Age doesn’t matter – as long as they change fields every ten years, they keep innovating.

    But, of course, who wants to publish with a relative newcomer?

    Another factor may be simply that very few people are brave (or foolhardy) enough to work on an idea which doesn’t already have lots of support. It could be interesting to look at the distribution curve of number of researchers doing early work on highly innovative ideas. I suspect the curve would slope upward toward zero like a Pareto distribution – which would imply that there’s lots of innovative ideas that no one is willing to work on, and if we want more innovation we need to change the incentive structure.

    1. MrXYZ says:

      Seymour Benzer’s career was a great example of this: solid state physics, phage genetics, genetic basis of circadian rythyms, structure of the brain, aging and neurodegenerative diseases. He was a busy man. Time, Love, Memory: A Great Biologist and His Quest for the Origins of Behavior but Jonathan Weiner is a great overview of Seymour Benzer’s scientific life. He appears to have changed fields about every 10 years.

  8. Kevin says:

    When considering PCR as an example of an exciting but garage-compatible discovery, consider also that PCR was developed in 1983. That’s thirty-six years ago now: near enough half a lifetime. (Or a full career–finish your Ph.D. at 29, retire at 65, and that’s your 36 years.)
    Impactful “garage-compatible” discoveries are not frequent happenings.

    (Consider also the number of working scientists now versus a century or two ago–the decline in the per-capita rate is brutal.)

  9. anon says:

    There are five basic personality traits: Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism (OCEAN). These traits are largely determined by genetics. Openness is associated with creativeness. Neither money nor man power can supply the genetic deficiency.

    1. bs_detector says:

      Asserts facts not in evidence.

  10. ChemBioJr says:

    This is an interesting point, that is very real to me at the moment. Due to my field of training and the nature of work I want to pursue in the future, I am finding that I have to be very picky when reviewing universities for a future junior faculty position. Which, we know, being “picky” with faculty jobs is something that is nearly impossible. It is more like, “be thankful you were considered by anyone”. But it would be futile accepting a position at a small uni that has no ability to support the work (w/ high tech instrumentation, etc) that I’ve been trained to do.. even if that is my only option for a job in academia.

    1. Anonymous says:

      I recently (Pipeline, Precision Medicine Real Soon Now, 31 January, 2019) mentioned FA Cotton’s lament that, even at TAMU, offers were not made to candidates that he felt were the best, brightest, and most creative. Offers went to candidates who would be fundable within existing NSF, NIH, DoE, DoD, or other well heeled program structures. Back then, it was buckyballs and fullerenes.

      Today, it could be Graphene, Photoredox, CO2 Reduction, … or the Kurrent Kudzu-du-jour (referencing Derek, from another Pipeline article).

      A famous chemist once told me that some ideas are “too academic even for academia.” 🙂 or should I say ;-(

      Even if your research program using start-up money is successful, if there is no way to obtain outside funding, you are toast. They will know that and not want to waste start-up money on you. If you want to find a place to fit in, SHOW THEM THE MONEY (funding streams that you will exploit).

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.