Nimbus Unavailable on Alamo

Submitted by Warren Smith on Sat, 31 Mar 2012, 15:49:38 GMT

FutureGrid Hardware Outage Information

Nimbus Unavailable on Alamo

Status
Resolved
Type
Software System
Impacted systems
alamo
Start of outage
Sat, 31 Mar 2012, 13:30 EDT
Anticipated end of outage
Thu, 12 Apr 2012, 22:00 EDT

Description

We're unfortunately having more problems with Nimbus on Alamo. Nimbus will be unavailable until we can resolve them.

Update: We are still working on these problems and hope to resolve them this week.

Update 2: Some of our tests are succeeding, but unfortunately, certain stress tests are causing node failures. We are upgrading to more recent versions of virtualization software (QEMU, KVM, libvirt) to attempt to resolve this issue. This upgrade has non-trivial dependencies and effects and will take us a little time to complete.

Resolution

Nimbus is available on Alamo. We apologize for the extended downtime while we worked through a number of issues. Please submit a ticket if you encounter any problems.

We upgraded the Nimbus nodes to CentOS 6 which allows us to use much newer versions QEMU, KVM, and libvirt. We upgraded to Nimbus 2.9 earlier during the downtime and we continue to use this version of Nimbus. These upgrades appear to have resolved the network crashes and kernel panics that were occurring under heavy Nimbus load.

»