In my most recent post, I outlined a model that allows estimation of National Institutes of Health (NIH) grant success rates based on the NIH appropriation history. This model has two components: a model that estimates the number of new and competing grants and a model that estimates the number of grant applications submitted to compete for funding. The term “grant”” in this case refers to NIH Research Project Grants (RPGs). This refers to grant mechanisms such as R01 and R21 grants that are the most common grants for individual or small groups of investigators. It excludes mechanisms for research centers and others that support larger groups of investigators. In this post, I present more details about the first component.
Historical appropriation and inflation data
To begin, I examine the appropriations history for NIH from 1990 to the present. NIH appropriations from 1990 to 2015 are shown below:
For comparison, success rates for grants (RPGs) are shown below:
These two parameters are negatively correlated with a correlation coefficient of -0.66. In other words, as the size of the appropriation increased, the success rate tended to decrease.
One possible adjustment that might improve the correlation involves correcting the appropriation data for inflation. Inflation is best measured in terms of the Biomedical Research and Development Price Index (BRDPI), a parameter calculated annually by the Department of Commerce on behalf of the NIH.
The NIH appropriation curves in nominal terms and in constant 1990 dollars are plotted below:
The constant dollar appropriations and success rate data are still negatively correlated with a correlation coefficient of -0.381.
Thus, the simple notion that the success rate should increase with increases in the NIH appropriation is empirically false over time.
There are two reasons why this is true. The first involves the manner in which NIH grants are funded. Grants average 4 years in duration which are almost always paid out in 4 consecutive fiscal years. Thus, if a 4-year grant is funded in a given fiscal year, the NIH is committed to paying the “out-years” for this grant over the next 3 fiscal years. Because of this, ~75% (actually more than 80% due to other commitments) of the NIH appropriation for a given year is already committed to ongoing projects, and only less than 20% of the appropriation is available for new and competing projects. This makes the size of the pool for new and competing projects very sensitive to the year-to-year change in the appropriation level.
The observed numbers of new and competing grants are plotted below:
A model for the annual number of new and competing grants
To put these effects in quantitative terms, a model has been developed to estimate the number of grants funded each year, given NIH appropriation and BRDPI data over time.
The assumptions used in building the model are:
- NIH funds grants with an average length of 4.0 years.
For the purposes of this model, we will assume 1/4 of the grants have a duration of 3 years, 1/2 of the grants have a duration of 4 years, and 1/4 of the grants have a duration of 5 years. Using a single pool of grants, all of which have 4-year durations, is both contrary to fact and would likely lead to artifacts. It is unlikely that the model will depend significantly on the details of the distribution. When a grant completes its last year, the funds are freed up to fund new and competing grants in the next year.
- The average grant size increases according to the BRDPI on a year-to-year basis.
This assumption has been less true in recent years owing to highly constrained NIH budgets, but this it a reasonable approximation (and still represents good practice).
- Fifty percent of the overall NIH appropriation each year is invested in RPGs. This is consistent with the average percentage of RPG investments over time.
- The system begins with an equal distribution of grants at each stage (first, second, … year of a multiyear grant) ~10 years before the portion used for analysis.
We will start in 1990. The comparison between the actual numbers of grants funding and those predicted by the model are shown below:
The agreement between the observed and predicted curves is remarkable. The correlation coefficient is 0.894.
Differences between the actual numbers of grants and those predicted by the model
The largest difference between the curves occurs at the beginning of the doubling period (1998-2003) where the model predicts a large increase in the number of grants that was not observed. This is due to the fact that NIH initiated a number of larger non–RPG-based programs when substantial new funding was available rather than simply funding more RPGs (although they did this to some extent). For example, in 1998, NIH invested $17 million through the Specialized Center–Cooperative Agreements (U54) mechanism. This grew to $146 million in 1999, $188 million in 2000, $298 million in 2001, $336 million in 2002, and $396 million in 2003. Note that the change each year matters for the number of new and competing grants that can be made because, for a given year, it does not matter whether funds have been previously committed to RPGs or to other mechanisms.
The second substantial difference occurs in 2013 when the budget sequestration led to a substantial drop in the NIH appropriation. To avoid having the number of RPGs that could be funded drop too precipitously, NIH cut noncompeting grants substantially. Noncompeting grants are grants for which commitments have been made and the awarding of a grant depends only on the submission of an acceptable progress report. The average size [in terms of total costs, that is, direct costs as well as indirect (facilities and administration) costs] of a noncompeting R01 grant was $393,000 in 2011, grew to $405,000 in 2012, a 2.9% increase, and then dropped to $392,000 in 2013, a 3.3% drop. Given that there are approximately three times as many noncompeting grants as there are new and competing grants, this change from a 2.9% increase to a 3.3% decrease for noncompeting grants increased the pool of funds for new and competing grants by ~3(2.9 + 3.3) = 18.6%. However, cutting noncompeting grants means that existing programs with research underway and staff in place had to find ways for dealing with unexpected budget cuts.
At this point, I have developed a reasonable model for estimating the number of new and competing awards that can be made given annual appropriation and BRDPI data. In the next post, I will examine a model for the number of grant applications submitted each year.
Available documents and code
The R Markdown file that generates this post, including the R code, is available.