Skip to Content

Editor's Blog , , , ,

NIH Impact Scores: Which Criteria Matter Most?

High-level administrators who blog are pretty uncommon, and high-level administrators who blog substantively are rarer still. That makes National Institute of General Medical Sciences (NIGMS) director Jeremy Berg a rare bird.

In recent days, Berg has posted to the NIGMS blog NIGMS Feedback Loop a three-part (so far) analysis of NIH’s new system for assigning scores to grant applications, in which reviewers assign a separate score for each of five core criteria. (The three blog posts are here, here, and here.) Berg has been analyzing the data to determine the correlation between each of the five criterion scores assigned by the reviewers and the overall impact score assigned by the study section. The analysis answers the question, which of the five criteria — significance, investigator, innovation, approach, and environment — matter most. The analysis is based on NIGMS data only, and so far from only one grant-reviewing cycle. Assuming the trends hold up, the results could be very useful for people applying for research grants from NIGMS. And assuming the trends apply across other NIH institutes — which remains to be seen — the results could be applicable to NIH grants generally.

The conclusions are mostly unsurprising, but they are also reassuring, especially for investigators who haven’t yet established reputations. Here’s Dr. Berg’s primary conclusion:

 This analysis indicates that approach and significance are the most
important factors, on average, in determining the overall impact score,
at least for this sample of NIGMS R01 grant applications.

“Approach,” with a Pearson correlation coefficient of 0.74, was the most closely correlated to the impact score; “significance” was next, at 0.63. The criteria least closely correlated with the impact score are “investigator” (0.49) and “environment” (0.37). That means that, at least for the sample Berg analyzed, the data show that the quality of your ideas matter far more than your reputation. That’s good news for early-career scientists.

Also notable: the weak correlation of “environment” with the impact score shows that where you are — your institution and the facilities it offers — matters even less than who you are.
On Twitter: @SciCareerEditor