Any organization that tries to make a wholesale upgrade of its key computer systems anticipates problems, but NIH’s electronic grant submission systems seem to have encountered more than their share over the past 3 weeks. As of this afternoon–12 June–NIH says its grant submission systems are working again, but the trail of messages to users during this time shows that the journey has been anything but easy.
On 19 May NIH told its electronic systems users that it planned a major upgrade for the end of May, costing $2-3 million. The upgrade promised to improve system performance and stability, offer better back-up in case of equipment failure, and provide an estimated 16 times its current capacity to handle future expansion. The agency closed the electronic submission functions from 22-26 May (a span that included the long Memorial Day holiday) for the upgrade. But when they tried to bring the systems back up, they ran into problems.
The scheduled relaunch on 27 May had to be pushed back a day while NIH worked through problems that a message to users called “residual issues.” When the systems finally did come back up in the afternoon of 28 May, users could not upload documents, such as reference letters, to the electronic Research Administration Commons (eRA Commons), the home base of NIH’s submissions functions. Later that afternoon, eRA Commons and its Internet Assisted Review systems (which offer critiques and preliminary scores on applications) had to be taken offline to fix more problems that developed.
On 29 May eRA Commons came back online and electronic submissions were being accepted–but the systems were working slower than normal and not all applications were processed correctly. Some users received system error messages. By 3 June, NIH had resolved the document upload problems, but the slow performance continued, apparently.
Yesterday (11 June), NIH decided to restart the systems, to resolve the performance problem. At 9:00 am, a message gave users 5 minutes to save their work and log-out of eRA Commons. What was expected to be a brief outage extended into the morning of 12 June, when eRA Commons came back up, and NIH announced that they “were able to isolate and fix the network/server issues that were the root cause” of the problems. But shortly after 10:00 am, NIH had to take the eRA Commons back down. By 2:00 pm, NIH was able to bring eRA Commons back up again, while closely monitoring its functions.
We’ll keep you posted on how NIH progresses with its electronic submissions, a topic we’ve been following almost from the beginning.