Skip to main content

Academia (vs. Industry)

Broader Impacts Indeed

Since I don’t have to write NSF grants, I haven’t had to wrestle with “Criterion 2”. But ask anyone in academic science about it. The first criterion is intellectual merit, as it darn well should be. Here’s the NSF’s own description (in full):

How important is the proposed activity to advancing knowledge and understanding within its own field or across different fields? How well qualified is the proposer (individual or team) to conduct the project? (If appropriate, the reviewer will comment on the quality of prior work.) To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts? How well conceived and organized is the proposed activity? Is there sufficient access to resources?

But the second criterion, while initially worthy-sounding, invites trouble. It’s “What are the broader impacts of the proposed activity?” Here’s more description:

How well does the activity advance discovery and understanding while promoting teaching, training, and learning? How well does the proposed activity broaden the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)? To what extent will it enhance the infrastructure for research and education, such as facilities, instrumentation, networks, and partnerships? Will the results be disseminated broadly to enhance scientific and technological understanding? What may be the benefits of the proposed activity to society?

To me, that puts the most important question last, and even that one can be hard to answer. As for the rest, this would seem to be an open invitation to insert all sorts of nice-sounding boilerplate, or to just start making things up. The NSF itself seems to have realized this, and has been working on a revised version of this language, but here’s a column from Dan Sarewitz that says that “Criterion 2.1” isn’t a bit better than the old one:

At the heart of the new approach is “a broad set of important national goals”. Some address education, training and diversity; others highlight institutional factors (“partnerships between academia and industry”); yet others focus on the particular goals of “economic competitiveness” and “national security”. The new Criterion 2 would require that all proposals provide “a compelling description of how the project or the [principal investigator] will advance” one or more of the goals.
The nine goals seem at best arbitrary, and at worst an exercise in political triangulation. . .Yet, more troubling than the goals themselves is the problem of democratic legitimacy. In applying Criterion 2, peer-review panels will often need to choose between projects of equal intellectual merit that serve different national goals. Who gave such panels the authority to decide, for example, whether a claim to advance participation of minorities is more or less important than one to advance national security?
. . .Motivating researchers to reflect on their role in society and their claim to public support is a worthy goal. But to do so in the brutal competition for grant money will yield not serious analysis, but hype, cynicism and hypocrisy.

One of the comments to that article points out that this isn’t the NSF’s fault, in a way, because this exact language was mandated by Congress. And so it is – take a look at Section 526 of Title V of H.R. 5116, the “America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science Reauthorization Act of 2010”. There’s all the same language. Not only that, but the Act directs the NSF to assign people and funds to evaluating how well all these “Broader Impact” measurements are going. The director, within six months, is supposed to have implemented a policy that:

. . .requires principal investigators applying for Foundation research grants to provide evidence of institutional support for the portion of the investigator’s proposal designed to satisfy the Broader Impacts Review Criterion, including evidence of relevant training, programs, and other institutional resources available to the investigator from either their home institution or organization or another institution or organization with relevant expertise.

So, in case you’ve lost track, the NSF is supposed to train people to implement a policy that requires grant applicants to show that their institutions are training people to implement a policy that requires grant applicants to show evidence that their work involves training people to implement a policy. I think I’ve got that right. A greater invitation to bullshit I cannot picture.

21 comments on “Broader Impacts Indeed”

  1. Cellbio says:

    OK, so perhaps the language here needs to be simplified, but lately I am working in an incubator where I see young kids getting exposure to science through an NSF grant explicitly for training. The program is put together with resources from industry, academic institutions and an investor network. From that perspective, the language of the second criterion seems quite clear to me as they require justification for the grant dollars meeting with other required elements. I am guessing the desire was to create language that enables funding of grants based upon scientific merit or training merit, but perhaps our sloppy systems now has grants of scientific (intellectual) merit also looking for points on the societal benefit side. But hasn’t that been the case for a long time? Read most any intro or conclusion to a paper and you’d think the work sets the stage for a cure next week. Heck, the Universities themselves now issue press releases making such claims.

  2. PharmaHeretic says:

    Pure kabuki theater
    This is more about glorifying the contemplation of research over the act of doing it. If research was always driven by such moronic requirements we would not have had almost all of the stuff we take for granted- and that includes almost EVERY single class of drugs.
    We simply cannot foresee where any particular research project will lead to. It is therefore better to have a 1000 easy to get small-to-medium sized grants and access to instrumentation over 50 highly funded sociopathic salesman aka ‘research superstars’.
    PS: The easiest way to turn any institution to crap is to make it “better” for political and legal types.

  3. Cartwheel says:

    That bit about one of the 2.1 goals being “diversity” reminds me of the recently-forgotten misstep by the current US administration of encouraging NASA to state that one of it’s prime goals was to “reach out to the Muslim World.”
    Yeah, like that’s going to get a spacecraft to Mars without crashing.
    Political Correctness is the death of us.
    “Diversity” is like “Clean Air.” Who could argue against it, right? Until you figure out that it is like you say, boilerplate.

  4. Flipflop says:

    Panels make recommendations, not decisions, based primarily on the science, swayed in part by notable (rare) broader impact parts of the proposal. NSF program officers make the decisions, and more criteria come into play, such as institutional diversity, PI status (ie new investigator), etc. The question of whether met your broader impacts objectives is more difficult to document than the science objectives, for which publications is the measure. So while more buerocracy will involve more red tape, it’s intended to address a difficult question- how much real results come from the broader impacts parts of NSF awards. I’d say this is addressed in annual and final grant reports, but question if NSF uses the reports to measure success for either science or broader impacts. Perhaps NSF could play a more useful role by guiding how broader impacts should be structured, their benefit evaluated, and documented at the end of a project.

  5. MTK says:

    As noted the wording and such is there to satisfy Congress. I doubt that any review committee is going to put nearly as much emphasis on that as Criterion 1.
    And yes, it just creates work for everyone to come up with the BS justifications which everyone, including Congress, is just a wink and a nod process.
    Is it a pain? Yes. Is it totally unnecessary? Yes. In an ideal world would we have to jump through these types of stupid hoops? No.
    But we do. Such is life.

  6. Lethe says:

    “Who gave such panels the authority to decide, for example, whether a claim to advance participation of minorities is more or less important than one to advance national security?”
    Congress did, as was amply demonstrated in the secnod part of this post. Who the hell else does that moron think did it? Magic fairies? Congress provides such authority all the goddamn time, and acting outraged and huffy about it is a preposterous pose.

  7. Melissasbench says:

    #4 Flipflop: “Perhaps NSF could play a more useful role by guiding how broader impacts should be structured, their benefit evaluated, and documented at the end of a project.”
    I wholeheartedly agree. There is intrinsic value in education and outreach programs, and there are thousands of kids that benefit from NSF-sponsored science programs. And it is noble and long-sighted to ask the actual scientists (instead of professional educators) to be involved in outreach and education.
    BUT — much of the problems come from asking researchers to INITIATE the outreach — to wear both hats at once. What constitutes appropriate Broader Impacts? How much can the PI take advantage of existing infrastructure for outreach and education, and how much needs to be completely independent of other initiatives? NSF needs to recommend specific modes of fulfilling Broader Impacts, in addition to soliciting thoughtful new programs. NSF is really passing the buck here to the PIs — Congress mandated it, NSF should figure out more specific means to achieve it instead of thrusting every PI that applies into a role for which he/she is untrained.
    I wholeheartedly believe basic scientists should be proactive in education and outreach. But we shouldn’t be required to each design our own programs, nor guess at what constitutes effective broader impacts. More examples, more specific guidelines please.

  8. Student says:

    I don’t understand the diversity requirement…Most of my classmates are classified as international and are definitely considered minorities. ????

  9. Student says:

    *To be more specific, at my institution 45% of the graduate students and 81% of the postdocs are international students.

  10. Rhenium says:

    Me thinks the lady doth protest too much…
    Despite the somewhat entitled nature of the post, Broader Impacts are 1) important and 2) hard to measure. Point 2 does not invalidate 1. Any grants office will tell you that if the intellectual merit doesn’t cut the mustard, it doesn’t matter how good your broader impacts are. All in all, I think this post shows that Derek hasn’t spent much time with NSF proposals.

  11. Anonymous says:

    Sorry to clutter the space here! Could someone do me a favor to proof read and edit this public letter? it will be send to the dean of science in Columbia!
    Should Mr. Sames apologize to these students being ruined by his fraud?
    Mr Sames is a tenured Professor in chemistry department of Columbia University, he focused on his study in organic chemistry. During the past 10 years, he is considered by peers as a genius chemist and published a variety of research publications in high impact journals such as JACS and Angew. However, there is a scandal breaking out in 2003 that one of his student Sezen (a female student from Turkey), who has done fraud in several publications and fabricated the research data, which in turn lead to her Phd degree being revoked from university of Columbia, a punish severe enough to an ex-student in graduate school. However, it seems there is no corresponding impact to Mr. Sames, we, a group of Chinese students (hundreds of Phd students and post-docs) in chemistry field hope there is a transparent investigation to clarify the result since there are several Chinese students being involved in this issue and being treated unfairly. Our perspectives toward this issue are very different with university of Columbia’s decision, and we are calling your attention to this issue;
    Below analysis regard Sames-Sezen issue is from ChemBark Blog, a neutral public blog for investigation of scientific misconduct
    ChemBark Investigations, Lab Management, Sames-Sezen, Scientific Culture, Scientific Misconduct.
    There are several reasons this section of the Columbia Report is particularly damning for Sames:
    1. Sames appears to punish several innocent students while allowing the guilty one to run wild.
    2. The action confirms that Sames knew that Sezen’s work could not be reproduced by several chemists in his own lab.
    3. The action suggests that rather than respond to the gravity of the irreproducibility and investigate it, Sames dismissed students from his lab (and in doing so, essentially silenced them).
    4. The fact that the Committee explicitly states that they were not charged with deciding whether Sames fired these students because of their inability to reproduce Sezen’s results raises a very important issue: while Sezen’s role in this scandal was investigated, we do not know whether Sames has been investigated. Furthermore, it is noteworthy that this investigation was launched as a result of a complaint by Sames against Sezen. Did Sames’ role as Complainant protect him from subsequent investigation?
    We know how Sezen has been punished, but what of Sames? Sezen is (rightly) going to lose her Ph.D. because it was based on fabricated data. Can Sames keep his job given his gross record of sustained negligence and the fact that a portion of Sezen’s work almost certainly factored in to Columbia’s decision to award him tenure in 2003?
    The answer is yes, apparently, because it has been six years since Sezen was exposed and Sames doesn’t appear to be going anywhere. While we don’t know how Columbia has punished Sames—if at all—we do know several punishments Sames has escaped. He still has his job at Columbia. He still has tenure. He still is allowed to run his own research lab. He still receives federal funding. He is still allowed to publish in JACS and seemingly was never suspended from doing so. Finally, he has evaded the vast majority of the acrimony surrounding the case. Somewhere between 2005 and 2011, the beautiful work of Sames became the horrible work of Sezen. How is it that a professor can be given the lionshare of credit for a body of good work when published, yet escape the lionshare of the blame when the work is proven fraudulent?”
    As above comments indicated, Mr Sames’qualification as a professor in a prestigious university is questionable, we Chinese chemistry community completely agree with above comments. In addition to Mr Sames’ behaviour in scientific research, the ethics of Mr Sames’s management of his lab and treatment of the students are also questionable; Indeed, we observed that he fired at least one of Chinese students in his group, maybe more, which bring a very tough time for them, and in turn attracted our Chinese chemists’ attention;
    Below is our Chinese chemistry community’s perspective regard to Sames-Sezen issue;
    It is a known fact that some professors take advantage of Indian and Chinese grad students and treat them as indentured servitude, just like some employers take advantage of illegal immigrants. It seems Mr. Sames is one of them. Indeed, Mr Sames slaved some of the best students in chemistry, one of the common sense of the characteristics of slavery, though, is the treatment of people as disposable objects, and certainly Sames treated some of his grad students as such.
    Mr. Sames hired the students from developing country and used them to explore the hiding landmines in the scientific field in their first two years in graduate school without proper training, then fired some of them, usually the one he dislike, would not flatter him, and being bombed by the random landmines in the scientific research field at their third year!
    Now days if a chemist want to work in his “beloved” chemistry field, he has to work like a slave, suck it up and go through it, hopefully with minimal amount of “permanent” damage. These are very true toward the students from developing countries in such an economic recession era. Most of schools realize that chemistry grad students probably are least likely candidates to donate a shining bldg or endow a chair. Compared to science graduate school, MD’s, JD’s and “know-it-all” MBA’s probably have very different grad school experience.
    In our perspective Mr Shames is one of the professors who ruined chemistry, in addition to that, he ruined US ‘s reputation as a leading, decent scientific research country. His behavior in scientific research and his ethic in management of his lab are unacceptable, which annoyed all the chemistry community. His conduct also brings very negative effects to future students majored in chemistry from rest of the world.
    There should be an investigation toward Mr Sames’ behavior and a corresponding punishment to his inappropriate deed in Columbia University!
    In such a recession time, Chinese chemistry community calls your attention regard this issue, regardless of your identity, either you are a student, being recently laid off or hold a tenure position, to protect our sensitive and fragile chemistry community.
    Chinese Chemistry Community in North America(CCCNA)

  12. Kieth says:

    “.. I think I’ve got that right. A greater invitation to bullshit I cannot picture.” Mr. Sarewitz may not have a future in academia but he lights up the page on that one.

  13. Anonymous says:

    While I’m not condoning Sames/Sezen, this is just one incident. I’m a bit concerned by the comments coming from “the chinese community”. How about the Chondroitin showing up in heparin API from china, lead based paint in toys shipped t the US and the list goes on!! You need to focus on YOUR problems over there….don’t focus on making trouble over here.

  14. Anonymous says:

    And I forgot, melamine in drinking milk!!! Someone was intentionally trying to increase the Nitrogen content to mimic protein content WTF!!! That is crime!

  15. cliffintokyo says:

    “…more hype, cynicism, and hypocrisy”
    Yep, we *definitely* need more of these (not).
    They are, after all, the staple fare of Congress, are they not?

  16. Chemist says:

    Honestly, whatever one writes and claims in his/her research proposal to get the funding, I believe research goals are very personal and does not have ethical/social strings attached to them. Why should I pay any tax for someone exploring Mars! I don´t give it a damn! and Why should one pay for me to see if my oragnocatalysis makes 5+2 cycloaddition or 3+2!

  17. Julia says:

    Maybe we should just eliminate all government funding of everything except roads and bridges.
    It would solve this problem, and it would solve the debt crisis.
    Besides, if someone gives you money, you still owe them something, even if it was a gift. More often than not, what you owe is your vote.

  18. Liz says:

    #5. MTK
    “I doubt that any review committee is going to put nearly as much emphasis on that as Criterion 1.”
    Have you ever applied for an NSF grant? Criterion 2 plays an equal role. For the pre-doc grants, it’s equally split between criterion 1 and criterion 2. Regular grants aren’t much different – you need both to be strong to get funded.

  19. none says:

    Have you ever applied for an NSF grant? Criterion 2 plays an equal role.
    HAHAHAHAHAHA!!!! It is to laugh. You obviously have never applied for an NSF grant, or at least not got the critiques back yet.

  20. cliffintokyo says:

    @17 Julia
    “Maybe we should just eliminate all government funding of everything except roads and bridges”
    Tempting oversimplification!
    Would not go down well here in Japan (Radiation and Tsunami devastated areas)
    There are always special cases.
    Its sometimes a tough job to distinguish between genuine special cases and *pork barrel* ones.

  21. MTK says:

    @Liz and None,
    Yes, I have applied for a NSF grant. It was granted also.
    I’m more in line with None’s thinking.

Comments are closed.