FutureGrid Outage Archive

Status	Type	Title	Impacted Systems	Start of Outage	Anticipated End of Outage	Resolution
Ongoing	Software System Hardware	Alamo cluster decommissioned	alamo	Wed, 01 Oct 2014, 08:06 EDT	Fri, 10 Oct 2014, 08:06 EDT
Resolved	Network	Alamo network outage this Saturday	alamo	Sat, 20 Sep 2014, 09:00 CDT	Sat, 20 Sep 2014, 17:00 CDT	Network maintenance was finished at 11:30 on 9/20.
Ongoing	Maintenance	Nimbus on Hotel has been shutdown	hotel	Fri, 19 Sep 2014, 17:00 EDT
Resolved	Network	OpenStack Hotel network outage	hotel	Thu, 18 Sep 2014, 17:00 EDT		The network issue has been resolved.
Resolved	Maintenance	Monthly maintenance Tuesday, September 2	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 02 Sep 2014, 08:00 EDT	Tue, 02 Sep 2014, 23:59 EDT	Maintenance has been completed and the systems have been returned to service. If you have a problem, please submit a ticket. --> http://archive.futuregrid.org/help
Planned	Software System	Eucalyptus on India	india	Tue, 19 Aug 2014 (All day)	Fri, 31 Oct 2014 (All day)
Resolved	Network	Hotel network changes	hotel	Tue, 12 Aug 2014, 07:00 CDT	Tue, 12 Aug 2014, 08:30 CDT	All changes have been completed with little to no interruption in service.
Resolved	Maintenance	Monthly maintenance Tuesday, 5 August	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 05 Aug 2014, 08:00 EDT	Tue, 05 Aug 2014, 23:59 EDT	Maintenance has been completed and the systems have been returned to service. If you have a problem, please submit a ticket. --> http://archive.futuregrid.org/help
Resolved	Maintenance Hardware	Hotel Maintenance	hotel	Tue, 29 Jul 2014, 08:00 CDT	Wed, 30 Jul 2014, 17:00 CDT
Resolved	Hardware	Nimbus/KVM on Hotel is unavailable	hotel	Fri, 18 Jul 2014, 13:30 EDT		There is still a network issue with Hotel/KVM.
Resolved	Hardware	Storage outage on Hotel	hotel	Wed, 16 Jul 2014, 08:30 EDT		Hotel is back online.
Resolved	Network	Network outage on Hotel	hotel	Wed, 09 Jul 2014, 17:30 EDT	Wed, 09 Jul 2014, 12:15 EDT	The network connectivity to Hotel has been restored. All services are available again.
Ongoing	Software System	Hotel compute going away	hotel	Wed, 09 Jul 2014, 16:00 CDT	Wed, 01 Oct 2014 (All day)
Resolved	Power	Unplanned outage - nas-0-0 power loss	sierra storage at SDSC	Mon, 07 Jul 2014, 14:25 PDT	Mon, 07 Jul 2014, 14:55 PDT	Power restored.
Resolved	Maintenance	Monthly maintenance Tuesday, 1 July	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 01 Jul 2014, 08:00 EDT	Tue, 01 Jul 2014, 23:59 EDT	Maintenance has been completed and the systems have been returned to service. If you have a problem, please submit a ticket. --> http://archive.futuregrid.org/help
Resolved	Software System	OpenStack Problems on Alamo	alamo	Mon, 16 Jun 2014, 12:00 EDT	Mon, 16 Jun 2014, 16:00 EDT	OpenStack services are operating correctly now.
Resolved	Software System	master node rebooted	alamo	Tue, 10 Jun 2014, 06:15 EDT	Tue, 10 Jun 2014, 09:08 EDT	Reboot fixed issue. Modifying LDAP logging options to prevent further issues.
Resolved	Maintenance	Various systems	other	Thu, 05 Jun 2014, 14:00 EDT	Mon, 16 Jun 2014 (All day)
Resolved	Maintenance	Maintenance on the Hotel Nimbus Xen cloud	hotel	Thu, 15 May 2014, 06:00 CDT	Thu, 15 May 2014, 07:00 CDT	The maintenance is over. The Hotel Nimbus Xen cloud is back online.
Resolved	Maintenance	Sierra scratch outage	sierra storage at SDSC	Tue, 13 May 2014, 14:00 PDT	Tue, 13 May 2014, 18:00 PDT	The maintenance on the scratch partition is complete.
Resolved	Network	Brief Sierra network outage	sierra storage at SDSC	Tue, 06 May 2014, 17:00 PDT	Tue, 06 May 2014, 19:00 PDT	Maintenance was completed w/o issues.
Resolved	Maintenance	Monthly maintenance Tuesday, 3 June	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 06 May 2014, 08:00 EDT	Tue, 06 May 2014, 23:59 EDT	Maintenance has been completed and the systems have been returned to service. If you have a problem, please submit a ticket. --> http://archive.futuregrid.org/help
Resolved	Maintenance	Monthly maintenance Tuesday, 6 May	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 06 May 2014, 08:00 EDT	Tue, 06 May 2014, 23:59 EDT	Maintenance has been completed and the systems have been returned to service. If you have a problem, please submit a ticket. --> http://archive.futuregrid.org/help
Resolved	Hardware	Sierra Outage	sierra	Fri, 11 Apr 2014, 09:00 EDT	Fri, 11 Apr 2014, 19:07 EDT	Outage was resolved.
Resolved	Software System	Alamo Open Stack outage	alamo	Tue, 08 Apr 2014, 20:03 CDT	Wed, 09 Apr 2014, 12:00 EDT	Openssl has been patched.
Resolved	Maintenance	Monthly maintenance Tuesday, 1 April	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 01 Apr 2014, 08:00 EDT	Tue, 01 Apr 2014, 23:58 EDT	Maintenance has been completed and the systems have been returned to service. If you have a problem, please submit a ticket. --> http://archive.futuregrid.org/help
Resolved	Software System	xray reserved	xray	Thu, 27 Mar 2014, 15:15 EDT	Wed, 09 Apr 2014, 17:00 EDT	Reservation concluded.
Resolved	Maintenance Network	Alamo network outage	alamo	Sat, 22 Mar 2014, 08:00 EDT	Sat, 22 Mar 2014, 17:00 EDT	Nework maintenance went as according to plan. Actual outage was only 2 hours.
Resolved	Maintenance	Monthly maintenance Tuesday, 4 March	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 04 Mar 2014, 08:00 EST	Tue, 04 Mar 2014, 23:59 EST	Maintenance has been completed and the systems have been returned to service. If you have a problem, please submit a ticket. --> http://archive.futuregrid.org/help
Resolved	Software System	India, bravo and delta are not available	india	Fri, 28 Feb 2014, 11:30 EST
Resolved	Software System	India is not available	india	Mon, 24 Feb 2014, 14:40 EST		Storage node was down, and we made the node back online and the issue should be resolved. If you still have an issue, please submit a ticket.
Resolved	Network	SDSC network outage scheduled for 2pm pacific today	sierra storage at SDSC	Thu, 13 Feb 2014, 14:00 PST	Thu, 13 Feb 2014, 14:10 PST
Resolved	Software System	Nimbus on Hotel inaccessible	hotel	Thu, 13 Feb 2014, 08:30 EST		The service has been restored.
Resolved	Software System	Eucalyptus on India	india	Wed, 12 Feb 2014 (All day)		A follow up not was made available that users can migrate to other systems
Resolved	Maintenance	OpenStack On India Unavailable	india	Mon, 10 Feb 2014, 21:00 EST	Tue, 11 Feb 2014, 18:00 EST
Resolved	Hardware	Sierra outage	sierra storage at SDSC	Sun, 09 Feb 2014, 03:35 PST	Mon, 10 Feb 2014, 09:05 PST	Node has been restored and services are backup.
Resolved	Software System	Sierra's IPoIB is not available	sierra	Tue, 04 Feb 2014, 12:00 EST
Resolved	Maintenance	Monthly maintenance Tuesday, 4 February	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 04 Feb 2014, 08:00 EST	Tue, 04 Feb 2014, 23:59 EST	Maintenance has been completed and the systems have been returned to service. If you have a problem, please submit a ticket. --> http://archive.futuregrid.org/help
Resolved	Software System	India is not available	india	Fri, 24 Jan 2014, 09:50 EST	Fri, 24 Jan 2014, 12:00 EST	We identified the issue and india is back online. If you still have a problem, please submit a ticket.
Resolved	Network	Alamo network slow connections	alamo	Mon, 20 Jan 2014, 06:05 EST	Wed, 22 Jan 2014, 14:00 EST	I upgraded the ntp package on all master, login and computes where applicable. I also added: restrict default kod nomodify nopeer notrap noquery options to the ntp.conf on all nodes.
Resolved	Network	Alamo network issue	alamo	Wed, 08 Jan 2014, 11:30 EST	Wed, 08 Jan 2014, 15:55 EST	network issue has been resolved.
Resolved	Maintenance	Monthly maintenance Tuesday, 7 January	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU	Tue, 07 Jan 2014, 08:00 EST	Tue, 07 Jan 2014, 23:59 EST	Maintenance has been completed and the systems have been returned to service. If you have a problem, please submit a ticket. --> http://archive.futuregrid.org/help
Resolved	Maintenance Network	Hotel network maintenance	hotel	Fri, 13 Dec 2013, 12:50 EST
Resolved	Power	SDSC power outage Dec 7 - 11	sierra storage at SDSC	Fri, 06 Dec 2013, 17:00 PST	Wed, 11 Dec 2013, 08:00 PST	Nimbus and Lima are back up.
Resolved	Maintenance Software System	Bravo is under maintenance Dec. 4	other	Wed, 04 Dec 2013, 09:00 EST	Wed, 04 Dec 2013, 23:59 EST
Resolved	Hardware	Sierra scratch and Nimbus offline	sierra storage at SDSC	Tue, 03 Dec 2013, 07:18 PST	Wed, 04 Dec 2013, 01:04 PST	We've restored the root partition and things are once again online. Sorry the for the inconvenience.
Resolved	Maintenance	Network Maintenance	india xray storage at IU network	Tue, 03 Dec 2013, 13:00 UTC	Tue, 03 Dec 2013, 14:00 UTC	Work completed as planned
Resolved	Maintenance	Monthly maintenance Tuesday, 3 December	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 03 Dec 2013, 08:00 EST		Maintenance has been completed and the systems have been returned to service. If you have a problem, please submit a ticket. --> http://archive.futuregrid.org/help
Resolved	Software System	Alamo LDAP issue	alamo	Fri, 15 Nov 2013, 12:47 EST	Wed, 20 Nov 2013, 23:00 EST	I restored the mysql database for the cluster management and we were able to sort out the LDAP configuration issues with alamo. All system services are available.
Resolved	Software System	Nimbus Hotel outage	hotel	Wed, 13 Nov 2013, 15:00 EST		Access to Nimbus has been restored.
Resolved	Network	India is unavailable	india storage at IU	Wed, 13 Nov 2013 (All day)	Wed, 13 Nov 2013, 11:30 EST	Restart sshd-lpk; reboot storage server.
Resolved	Maintenance	Sierra outage	sierra storage at SDSC	Fri, 08 Nov 2013, 17:30 PST	Fri, 08 Nov 2013, 23:59 PST	updates complete.
Resolved	Software System	Openstack/Alamo Full	alamo	Thu, 07 Nov 2013, 12:00 EST	Mon, 11 Nov 2013, 18:00 EST	We haven't added additional nodes to OpenStack yet, but there are resources free to run virtual machines now. We still plan to add additional nodes to OpenStck in the near future.
Resolved	Maintenance	Monthly maintenance Tuesday, 5 November	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 05 Nov 2013, 08:00 EST	Tue, 05 Nov 2013, 23:59 EST	Maintenance has been completed and the systems have been returned to service. If you have a problem, please submit a ticket. --> http://archive.futuregrid.org/help
Resolved	Software System	OpenStack on Alamo down	alamo	Wed, 30 Oct 2013, 17:00 EDT	Wed, 30 Oct 2013, 19:15 EDT	OpenStack is available again.
Resolved	Maintenance	OpenStack on Alamo Downtime	alamo	Tue, 29 Oct 2013, 09:00 EDT	Tue, 29 Oct 2013, 17:00 EDT	Maintenance is complete.
Resolved	Maintenance Network	TACC network outage	alamo	Sat, 26 Oct 2013, 08:00 EDT	Sat, 26 Oct 2013, 18:00 EDT	The network maintenance was completed by mid-afternoon.
Resolved	Hardware	XRAY is unavailable	xray	Thu, 17 Oct 2013 (All day)	Tue, 22 Oct 2013, 08:30 EDT	Replaced shorted VRM on node c0-0c1s3.
Resolved	Hardware	Sierra nimbus and scratch outage	storage at SDSC	Tue, 15 Oct 2013, 10:35 PDT	Tue, 15 Oct 2013, 11:16 PDT	Storage is back up.
Resolved	Hardware	Sierra nimbus and scratch outage	storage at SDSC	Tue, 08 Oct 2013, 08:52 PDT	Tue, 08 Oct 2013, 13:40 PDT	Machine is backup.
Resolved	Hardware	Sierra nimbus and scratch outage	storage at SDSC	Tue, 08 Oct 2013, 08:45 PDT	Fri, 11 Oct 2013, 10:00 PDT	Machine is back up
Resolved	Software System	Nimbus on Sierra is unavailable	sierra	Thu, 03 Oct 2013, 15:00 EDT		Access to Nimbus Sierra has been restored.
Resolved	Maintenance	Nimbus maintenance on Alamo	alamo	Tue, 01 Oct 2013, 08:00 EDT	Wed, 02 Oct 2013, 18:00 EDT	Fixed certificate problems Added support for qcow2 images. Improved networking performance to VMs.
Resolved	Maintenance	Monthly Maintenance Tuesday, 1 October	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 01 Oct 2013, 08:00 EDT	Tue, 01 Oct 2013, 23:59 EDT	Maintenance has been completed and the systems have been returned to service. If you have a problem, please submit a ticket. --> http://archive.futuregrid.org/help
Resolved	Software System	Sierra OpenStack: euca2ools is not available	sierra	Wed, 25 Sep 2013, 09:00 EDT
Resolved	Software System	Bravo: 5 week-reservation by FG-366, starting on Sep. 23	other	Mon, 23 Sep 2013, 09:00 EDT	Mon, 11 Nov 2013, 12:00 EST
Resolved	Hardware	Sierra is not available	sierra storage at SDSC	Tue, 17 Sep 2013, 15:00 EDT		Resilvering has been finished and Sierra is back online. If you still have an issue, please submit a ticket.
Resolved	Software System	xray reserved through 27-SEP-2013	xray	Fri, 13 Sep 2013, 17:30 EDT	Fri, 27 Sep 2013, 17:00 EDT
Resolved	Network	Network outage	alamo	Mon, 09 Sep 2013, 12:30 CDT	Mon, 09 Sep 2013, 22:00 EDT	There was an issue with a compute node that was causing a routing loop on the Alamo network. I believe we have found the issue, network seems to be stable now. All services are available again.
Resolved	Hardware	Periodic issues with Sierra storage affecting Nimbus and scratch storage	storage at SDSC	Mon, 09 Sep 2013, 01:08 EDT	Thu, 19 Sep 2013, 08:40 PDT	Update 9/19: We believe this problem has been resolved (i.e., was a false positive) and have not observed any new issues with the boot disk since 9/10.
Resolved	Network	Alamo Unavailable	alamo	Sat, 07 Sep 2013, 09:00 EDT	Mon, 09 Sep 2013, 09:30 EDT	A switch was replaced on the Alamo network. Openstack, Nimbus and HPC partitions are available again.
Resolved	Hardware	Outage on Sierra's storage for Nimbus and Openstack	sierra storage at SDSC	Thu, 05 Sep 2013, 06:00 PDT	Fri, 06 Sep 2013, 15:00 PDT	Problem has been resolved and service has been restored
Resolved	Maintenance	Monthly Maintenance Tuesday, 3 September	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 03 Sep 2013, 08:00 EDT	Tue, 03 Sep 2013, 23:59 EDT	Maintenance has been completed and the systems have been returned to service. Please send email to help@tickets.futuregrid.org for any issues.
Resolved	Power	foxtrot unavailable for building electrical power maintenance	foxtrot	Sat, 31 Aug 2013, 06:00 EDT	Sat, 31 Aug 2013, 20:00 EDT	Electrical power upgrade completed.
Resolved	Power	Alamo outage	alamo	Tue, 27 Aug 2013, 15:30 CDT	Tue, 27 Aug 2013, 17:00 CDT	Some nodes need to be be rebooted and some services need to be restarted.
Resolved	Network	Sierra is not available right now with some issue on DNS	sierra	Wed, 21 Aug 2013, 09:00 EDT	Wed, 21 Aug 2013, 23:59 EDT	Sierra's entries on IU DNS are fixed.
Resolved	Software System	Alamo SSH and login problems	alamo	Thu, 08 Aug 2013 (All day)	Fri, 09 Aug 2013 (All day)
Resolved	Software System	Storage, India currently unavailable	india storage at IU	Sat, 03 Aug 2013, 13:44 EDT	Sun, 04 Aug 2013, 11:00 EDT	Storage services are now available.
Resolved	Software System Hardware	Limited support through August 20	india foxtrot sierra xray storage at IU	Thu, 01 Aug 2013 (All day)	Mon, 19 Aug 2013 (All day)
Resolved	Software System	Eucalyptus on India is not available	india	Wed, 31 Jul 2013, 09:00 EDT	Wed, 31 Jul 2013, 23:59 EDT	Eucalyptus is operating normally.
Resolved	Software System	India's HPC partition is reserved July 29 - August 6 (9 days)	india	Mon, 29 Jul 2013, 08:00 EDT	Tue, 06 Aug 2013, 20:00 EDT	Reprovisioning of HPC image is done. If you still have a problem using india's job scheduler, please submit a ticket.
Resolved	Hardware	Sierra storage outage	sierra storage at SDSC	Sun, 21 Jul 2013, 21:05 EDT	Mon, 22 Jul 2013, 09:00 PDT	Sierra's scratch and /images is back online.
Resolved	Maintenance	Crowd service offline for security update.	other	Tue, 16 Jul 2013, 15:30 EDT	Tue, 16 Jul 2013, 18:00 EDT	Update completed.
Resolved	Maintenance Network	Network outage affecting IU resources	india xray network other	Sun, 14 Jul 2013, 09:00 EDT	Sun, 14 Jul 2013, 10:00 EDT	The router firmware upgrade was completed as planned.
Resolved	Power	Indiana University resources may be unstable	india xray other	Wed, 10 Jul 2013, 16:30 EDT	Wed, 10 Jul 2013, 23:00 EDT
Resolved	Power	Xray is down due to power failure	xray	Wed, 10 Jul 2013, 16:30 EDT	Wed, 10 Jul 2013, 23:00 EDT	Xray is back online
Resolved	Software System	Nimbus is unavailable on Hotel	hotel	Mon, 08 Jul 2013, 10:30 EDT		Issue is resolved.
Resolved	Software System	OpenStack Essex on india has issue on Nova Volume Service.	india	Mon, 08 Jul 2013, 09:00 EDT	Wed, 31 Jul 2013, 09:00 EDT	We applied unofficial workaround patches to all the nova compute nodes, and restarted the service. Now it is fine to create/attach volumes. But we still recommend Sierra's OpenStack for the users who need to use volume service, until we upgrade india's.
Resolved	Maintenance	Monthly Maintenance Tuesday, 2 July	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 02 Jul 2013, 08:00 EDT	Tue, 02 Jul 2013, 23:59 EDT	All systems are back to normal. Please report any issues by submitting a ticket.
Resolved	Network	Alamo networking outage	alamo	Fri, 21 Jun 2013, 16:12 EDT	Fri, 21 Jun 2013, 22:00 CDT	TACC networking issue has been resolved. Please report any issues or concerns.
Resolved	Software System	Nimbus on Hotel is unavailable	hotel	Tue, 18 Jun 2013, 15:30 EDT	Tue, 18 Jun 2013, 17:00 EDT	Nimbus has been restored.
Resolved	Hardware	Sierra outage	sierra storage at SDSC	Thu, 13 Jun 2013, 07:56 PDT	Thu, 13 Jun 2013, 08:21 PDT	Storage is back online.
Resolved	Maintenance	Monthly Maintenance Tuesday, 4 June	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 04 Jun 2013, 06:00 EDT	Tue, 04 Jun 2013, 23:59 EDT
Resolved	Maintenance	Foxtrot, India, Sierra and Hotel will be used for Tutorial Session.	india foxtrot sierra hotel	Sun, 02 Jun 2013, 20:00 EDT	Mon, 03 Jun 2013, 05:30 EDT
Resolved	Network	Alamo networking outage	alamo	Sat, 18 May 2013, 09:00 EDT	Sun, 19 May 2013, 17:00 EDT	Network maintenance complete. Monitoring status and tickets for any residual issues.
Resolved	Software System	Bravo is reserved for 10 weeks	other	Tue, 14 May 2013, 17:00 EDT	Mon, 29 Jul 2013, 17:00 EDT
Resolved	Maintenance	Firmware upgrade on Sierra storage	sierra storage at SDSC	Fri, 10 May 2013, 13:30 PDT	Fri, 10 May 2013, 14:49 PDT	Firmware upgrades have been completed
Resolved	Software System	Delta is not available	other	Tue, 07 May 2013, 23:00 EDT	Wed, 08 May 2013, 17:00 EDT	Delta is back online. Please submit a ticket if you still have a problem.
Resolved	Maintenance	Monthly Maintenance Tuesday, 7 May	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 07 May 2013, 08:00 EDT	Tue, 07 May 2013, 23:59 EDT	All systems are back to normal. Please report any issues by submitting a ticket.
Resolved	Maintenance Software System	Nimbus monthly maintenance	sierra hotel	Tue, 07 May 2013, 05:30 EDT		The Nimbus maintenance has been successfully performed.
Resolved	Hardware	Nimbus unavailable on Sierra	sierra	Sun, 05 May 2013, 10:00 EDT		Nimbus is back online.
Resolved	Hardware	Hotel is unavailable	hotel	Sun, 28 Apr 2013, 11:00 EDT		Hotel is back online.
Resolved	Hardware	Nimbus offline on Sierra	sierra	Tue, 23 Apr 2013, 15:43 PDT		Nimbus is back online.
Resolved	Hardware	Sierra storage offline	sierra storage at SDSC	Tue, 23 Apr 2013, 15:43 PDT	Wed, 24 Apr 2013, 07:43 PDT	Storage is now online. We are expecting some replacement disk hopefully today that have been on backorder. Please backup any important data.
Resolved	Software System	Nimbus on Sierra is unavailable due to a reconfiguration	sierra	Wed, 10 Apr 2013, 05:00 EDT	Wed, 10 Apr 2013, 09:45 EDT	LANTorrent is running now on Sierra.
Resolved	Software System	Nimbus on Sierra will be unavailable due to a reconfiguration	sierra	Mon, 08 Apr 2013, 06:00 EDT	Mon, 08 Apr 2013, 19:00 EDT	Due tu unforeseen complications, Sierra is still running with the default scp propagation.
Resolved	Hardware Power	India, bravo and delta are not available.	india	Thu, 04 Apr 2013, 18:00 EDT	Fri, 05 Apr 2013, 09:00 EDT	The system is back. If you still have a problem, please submit a ticket.
Resolved	Maintenance	Monthly Maintenance Tuesday, 2 April	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 02 Apr 2013, 05:00 EDT	Tue, 02 Apr 2013, 23:59 EDT	All systems are back to normal. Please report any issues by submitting a ticket.
Resolved	Software System	Eucalyptus on India is not available	india	Tue, 26 Mar 2013, 09:00 EDT	Tue, 26 Mar 2013, 23:59 EDT	The euca-nc nodes were having an issue on communicating with ldap server. We resolved it and restart euca-nc service. If you still have a problem, please submit a ticket.
Resolved	Software System	Nimbus outage	alamo	Wed, 13 Mar 2013, 06:00 EDT	Wed, 13 Mar 2013, 06:05 EDT	Nimbus partition available. My client was not able to connect because of configuration error on my end. No real outage after the service was moved: http://archive.futuregrid.org/news/nimbus-upgrade-alamo
Resolved	Software System	OpenStack on india is not available	india	Fri, 08 Mar 2013, 10:00 EST		Floating IP addresses were not released properly. We restart OpenStack and apply latest patches. If you still have a problem, please submit a ticket.
Resolved	Software System	Nimbus is not available on Sierra	sierra	Fri, 08 Mar 2013, 10:00 EST		The login problems on VMM nodes have been solved, Nimbus is back online.
Resolved	Software System	India and sierrra are not available.	india sierra	Fri, 08 Mar 2013, 10:00 EST		We found out the communication failure between ldap server and compute nodes, and now it's fixed. Please submit a ticket if you still have a problem.
Resolved	Maintenance	Addtional maintenance on India, Bravo and Delta	india	Wed, 06 Mar 2013 (All day)	Wed, 06 Mar 2013, 23:59 EST	Additional maintenance work is done. Please report any issues by submitting a ticket.
Resolved	Maintenance	Monthly maintenance Tuesday, 5 March	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 05 Mar 2013, 08:00 EST	Tue, 05 Mar 2013, 23:59 EST	Maintenance is mostly done. We need additional maintenance work for india, bravo and delta for tomorrow.
Resolved	Software System	Nimbus upgrade to 2.10.1 on Hotel	hotel	Tue, 05 Mar 2013, 04:00 EST	Tue, 05 Mar 2013, 23:59 EST	Nimbus has been updated and is running correctly. Unfortunately due to a mistake during the upgrade process SSH keypairs upload through the EC2 API have been deleted. If you use this functionality you will have to upload your keys again. Sorry for the inconvenience.
Resolved	Software System	Nimbus Unavailable on Sierra	sierra	Thu, 28 Feb 2013, 13:00 EST	Thu, 28 Feb 2013, 18:00 EST	The disk problems have been resolved.
Resolved	Software System	Nimbus Unavailable on Alamo	alamo	Wed, 27 Feb 2013 (All day)	Tue, 05 Mar 2013, 23:00 EST	Nimbus on Alamo has been upgraded to version 2.10.1. ** The location of the Nimbus services has changed: Users must edit their nimbus-cloud-client-0XX/conf/alamo.conf to continue using Nimbus on Alamo. All occurrences of 'master1' must be replaced with 'nimbus' in the alamo.conf file. We apologize for this inconvenience and for the extended downtime. For changes in this version, please see the change log on the Nimbus web site (http://www.nimbusproject.org/docs/2.10.1/changelog.html)
Resolved	Software System	Eucalyptus on india restarted	india	Mon, 11 Feb 2013, 10:20 EST	Mon, 11 Feb 2013, 10:30 EST	The service was restarted and it's back online.
Resolved	Maintenance	Nimbus outage on Sierra	sierra	Thu, 07 Feb 2013, 15:00 EST	Thu, 07 Feb 2013, 17:00 EST	The disk has been successfully replaced.
Resolved	Network	Brief Sierra network outage	sierra storage at SDSC	Thu, 07 Feb 2013, 09:00 PST	Thu, 07 Feb 2013, 09:15 PST	Maintenance completed.
Resolved	Maintenance	Monthly maintenance Tuesday, 5 February	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 05 Feb 2013, 08:00 EST	Tue, 05 Feb 2013, 23:59 EST
Resolved	Network	SDSC Networking outage	sierra storage at SDSC	Fri, 01 Feb 2013, 15:00 PST	Fri, 01 Feb 2013, 15:30 PST	Update was completed.
Resolved	Software System	Alamo unavailable this morning	alamo	Wed, 09 Jan 2013, 06:49 EST	Wed, 09 Jan 2013, 08:05 EST	I cleared out the file and restarted LDAP. Alamo is back in production.
Resolved	Maintenance	Monthly maintenance Tuesday, 8 January	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 08 Jan 2013, 08:00 EST	Tue, 08 Jan 2013, 23:59 EST	All systems are back to normal. Please report any issues by submitting a ticket.
Resolved	Hardware	Storage outage on Sierra	sierra storage at SDSC	Thu, 03 Jan 2013, 09:00 PST	Thu, 03 Jan 2013, 15:11 PST	The storage machine is back online for now. We are still looking into some problems with it but data should once again be accessible.
Resolved	Software System	Nimbus outage on Sierra	sierra	Mon, 24 Dec 2012 (All day)	Sat, 29 Dec 2012, 14:30 EST
Resolved	Network	Planned network outage at SDSC	sierra storage at SDSC	Tue, 18 Dec 2012, 23:00 PST	Tue, 18 Dec 2012, 23:24 PST	Maintenance was completed at 11:24PM.
Resolved	Software System	Xray upgrade	xray	Mon, 17 Dec 2012, 09:00 EST		18-DEC-2012 Status: one of the L0 controllers is not responding after a BIOS reflash. Case has been filed with Cray. 24-DEC-2012 Status: several steps in the upgrade are taking much longer than anticipated. Progress continues but time to return Xray to service is currently unknown. We will continue to update status here. 04-JAN-2013 Xray is now available again. Please report any issues to help@futuregrid.org. Summary of what's new/available: - Cray Linux Environment (CLE) 3.1UP03A - Cray Cluster Compatibility Mode (CCM) - CADE (Cray Application Developers Envirionment) 6.10. - Portland Group (PGI) Compilers 12.10 - Cray Compiler Environment (CCE) 7.4.7 - CRAYPAT (Performance Analysis Tools) 5.0.2 - Torque 2.4.14 and MOAB 5.3.6
Resolved	Hardware	Sierra storage offline	sierra storage at SDSC	Fri, 14 Dec 2012, 15:00 PST	Mon, 17 Dec 2012, 02:54 PST	We've replaced a few disks to get the storage back online. A few more have disks have failed but we currently have enough redundancy and we expect to get more disk replacements this week.
Resolved	Software System	RAIN software maintenance	india	Fri, 14 Dec 2012, 10:35 EST	Fri, 14 Dec 2012, 13:00 EST	The maintenance has been completed and the service is back online.
Resolved	Maintenance	Monthly maintenance Tuesday, 4 December	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 04 Dec 2012, 08:00 EST	Tue, 04 Dec 2012, 23:59 EST	All systems are back to normal. Please report any issues by submitting a ticket.
Resolved	Software System	Inca outage	other	Thu, 22 Nov 2012, 06:50 EST	Sat, 24 Nov 2012, 01:00 EST	Machine is back up and services have been restarted.
Resolved	Network	Alamo outage	alamo	Mon, 19 Nov 2012, 12:30 EST	Mon, 19 Nov 2012, 15:07 EST	User was stressing a the NFS /home file system. User has been contacted and NFS server is stable again.
Resolved	Software System	Eucalyptus on india is down	india	Mon, 19 Nov 2012, 11:45 EST	Mon, 19 Nov 2012, 17:15 EST	The issue might be related to a known bug - https://eucalyptus.atlassian.net/browse/EUCA-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel. At this time we just managed to restore the service and will possible apply the patch or upgrade the system later.
Resolved	Maintenance	india: compute nodes reallocation	india	Fri, 09 Nov 2012, 10:00 EST	Sat, 10 Nov 2012, 23:59 EST
Resolved	Maintenance	Monthly maintenance Tuesday, 6 November	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 06 Nov 2012, 08:00 EST	Tue, 06 Nov 2012, 23:59 EST
Resolved	Software System	Ticket system down	other	Mon, 05 Nov 2012, 12:00 EST	Mon, 05 Nov 2012, 19:30 EST	After a reboot and lengthy fsck, the ticket system is back online.
Resolved	Hardware	xray login inaccessible	xray	Mon, 05 Nov 2012, 10:00 EST	Tue, 06 Nov 2012, 10:16 EST	xray was bounced to resolve this issue as well as re-enable a disabled compute node. System is operational again now.
Resolved	Maintenance	Eucalyptus maintenance	india	Sat, 03 Nov 2012, 09:00 EDT	Sun, 04 Nov 2012, 23:59 EST
Resolved	Hardware	[resolved] openstack is not available	india	Sat, 03 Nov 2012, 00:45 EDT	Mon, 05 Nov 2012, 09:00 EST	The management node was down by disk failure. We replaced the disk and rebuilt Nova, and OpenStack is back online. During this recovery, your ssh keys are missing. So you need to reimport or recreate your key again, by the following commands. import exisiting key: euca-import-keypair -f .ssh/id_rsa.pub yourkeyname add new key: euca-add-keypair yourkeyname > filename.pub
Resolved	Software System	Nimbus outage	hotel	Fri, 02 Nov 2012 (All day)	Fri, 02 Nov 2012, 14:30 EDT	Nimbus is back.
Resolved	Software System	Eucalyptus on india restarted	india	Mon, 29 Oct 2012, 10:40 EDT	Mon, 29 Oct 2012, 10:48 EDT	The service is restarted and it's back to operation.
Resolved	Software System	Eucalyptus on india restarted	india	Mon, 22 Oct 2012, 10:35 EDT	Mon, 22 Oct 2012, 10:40 EDT	The service has been restarted and it went back online.
Resolved	Software System	openstack is not available	india	Wed, 17 Oct 2012, 21:00 EDT	Thu, 18 Oct 2012, 12:00 EDT	The database for nova component was having a serious issue, and it would take long fixing the tables one by one. So we restarted all service and rebuilt the nova database. So OpenStack is available now, but user will need to run the following euca comand to import ssh key again. euca-import-keypair -f .ssh/id_rsa.pub your_keyname Sorry for the inconvenience.
Resolved	Maintenance Hardware	Xray Hardware Maintenance	xray	Mon, 15 Oct 2012, 08:00 EDT	Mon, 15 Oct 2012, 17:00 EDT
Resolved	Maintenance	Campus cooling outage: Alamo	alamo	Sat, 13 Oct 2012, 17:00 EDT	Mon, 15 Oct 2012, 10:00 EDT	Cooling system repair was successful.
Resolved	Software System	Alamo outage	alamo	Thu, 11 Oct 2012, 12:00 EDT	Thu, 11 Oct 2012, 02:30 EDT	files system cleaned up. LDAP service restarted.
Resolved	Software System	india's strage is down	india storage at IU	Thu, 11 Oct 2012, 04:00 EDT	Thu, 11 Oct 2012, 12:00 EDT	The storage was down by overload, and we rebooted the storage and it got back fine.
Resolved	Software System	Eucalyptus on sierra is not available	sierra	Wed, 10 Oct 2012, 09:00 EDT		We finished the replacement. OpenStack Grizzly on Sierra is now available for early users. If you're interested in trying Grizzly. Please submit a ticket.
Resolved	Software System	Eucalyptus on india is not available	india	Tue, 09 Oct 2012, 09:00 EDT	Wed, 10 Oct 2012, 09:00 EDT	Eucalyptus compute nodes were having some issues. We restarted these nodes, which fixed the issue.
Resolved	Maintenance	Hotel maintenace	hotel	Tue, 09 Oct 2012, 07:00 CDT	Tue, 09 Oct 2012, 23:00 CDT	Monthly maintenance is done.
Resolved	Software System	Eucalyptus on india restarted	india	Wed, 03 Oct 2012, 17:20 EDT	Wed, 03 Oct 2012, 17:28 EDT	The service has been restarted and it's back to normal operation
Resolved	Software System	alamo login node rebooted today	alamo	Wed, 03 Oct 2012, 13:12 CDT	Wed, 03 Oct 2012, 13:59 EDT	reboot sorted out the issue.
Resolved	Software System	Upgrading Nimbus on Sierra	sierra	Tue, 02 Oct 2012, 12:00 EDT	Tue, 02 Oct 2012, 16:00 EDT
Resolved	Software System	Eucalyptus on india restarted	india	Mon, 01 Oct 2012, 17:05 EDT	Mon, 01 Oct 2012, 17:15 EDT	The service has been restarted and back to operation.
Resolved	Network	SDSC FutureGrid network outage	sierra storage at SDSC	Fri, 28 Sep 2012, 22:00 PDT	Sat, 29 Sep 2012, 04:00 PDT	Completed.
Resolved	Software System	Sierra Nimbus Upgrade	sierra	Fri, 28 Sep 2012, 11:30 EDT	Fri, 28 Sep 2012, 15:30 EDT	Nimbus upgrade is postponed until Sierra storage problems are resolved.
Resolved	Hardware	Sierra partial storage outage	sierra storage at SDSC	Thu, 27 Sep 2012, 15:01 PDT	Tue, 02 Oct 2012, 16:00 PDT	All nodes were rebooted on 10/1 and the Nimbus install has also been updated today. From some initial tests, everything seems to be working. Please file a ticket with help@futuregrid.org if anything seems amiss.
Resolved	Hardware	Sierra disk outage	storage at SDSC	Mon, 24 Sep 2012, 10:35 PDT		The storage box is back online for now. Two failed disks have been replaced and ZFS is still working to integrate back into the pool but since the filesystem is Raid2, all files are available and Sierra is back online. Further maintenance may be needed over the next few days.
Resolved	Network	Sierra network outage	sierra storage at SDSC	Tue, 18 Sep 2012, 18:00 PDT	Tue, 18 Sep 2012, 20:00 PDT	Maintenance has been complete.
Resolved	Network	Network Problems for India and Sierra	india sierra	Tue, 18 Sep 2012, 12:00 EDT	Tue, 18 Sep 2012, 12:30 EDT	The problem was that in one of the essential machines the root partition was full. This prevented the DHCP server to properly start.
Resolved	Software System Network	OpenStack Outage	india	Mon, 17 Sep 2012, 15:00 EDT
Resolved	Software System Network	Eucalyptus 3 Outage on Sierra and India	india sierra	Mon, 17 Sep 2012, 15:00 EDT	Mon, 17 Sep 2012, 21:00 EDT
Resolved	Network Power	Alamo unavailable after power outage	alamo	Fri, 14 Sep 2012, 12:45 EDT	Mon, 17 Sep 2012, 23:54 EDT	The networking issue has been resolved. A networking change was promoted when we experience the power outage.
Resolved	Software System	Hotel Nimbus Upgrade	hotel	Fri, 14 Sep 2012, 04:00 EDT	Fri, 14 Sep 2012, 06:00 EDT
Resolved	Software System	Eucalyptus on Inida was down and back to service	india	Fri, 14 Sep 2012 (All day)	Fri, 14 Sep 2012, 11:59 EDT	The service has been restarted.
Resolved	Network	Sierra network outage	sierra	Thu, 13 Sep 2012, 14:30 PDT	Thu, 13 Sep 2012, 15:51 PDT	The updates have been completed.
Resolved	Maintenance Software System	OpenStack restart	india	Wed, 12 Sep 2012, 17:30 EDT	Wed, 12 Sep 2012, 17:50 EDT	The OpenStack nova system was restarted on India. Some VMs may have been lost.
Resolved	Software System	euca-attach-volume is not available on OpenStack	india	Fri, 07 Sep 2012, 22:00 EDT	Sun, 30 Sep 2012, 23:59 EDT	The bug is not officially fixed yet, so we tested a workaround patch and it worked fine. So we applied it on the production.
Resolved	Software System	India's OpenStack is not available	india	Fri, 07 Sep 2012, 22:00 EDT	Sun, 09 Sep 2012, 23:59 EDT	We applied the latest patches and the problem has been resolved.
Resolved	Software System Hardware	Storage system down	india storage at IU network	Sun, 02 Sep 2012, 21:07 EDT	Sun, 02 Sep 2012, 23:00 EDT	Restored service and access service to India, Bravo, and Delta. HPC queues should be restored soon.
Resolved	Software System	Eucalyptus GUI not available	india	Sat, 01 Sep 2012 (All day)	Sun, 05 Oct 2014 (All day)	the gui will not be supported on india. a follow up not was created on the status pages to migrate. On this new systems the GUI may also not be available. Users can use for example cloudmesh
Resolved	Hardware	India's Openstack is not available	india	Mon, 20 Aug 2012, 17:35 EDT	Tue, 21 Aug 2012, 09:00 EDT	We resolved the disk failure, checked and repaired the file system.
Resolved	Maintenance	Monthly maintenance Tuesday, 7 August	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 07 Aug 2012, 08:00 EDT	Tue, 07 Aug 2012, 23:59 EDT	All systems are back to normal. Please report any issues by submitting a ticket.
Resolved	Network	Networking Outage Alamo	alamo	Tue, 07 Aug 2012, 08:00 EDT	Tue, 07 Aug 2012, 03:15 EDT	Alamo network successfully moved to higher capacity, redundant router hardware.
Resolved	Software System Hardware	Science Cloud Summer School	india foxtrot sierra hotel xray alamo	Mon, 30 Jul 2012, 09:00 EDT	Fri, 03 Aug 2012, 19:00 EDT	summer school ending today.
Resolved	Software System	Eucalyptus maintenance	india	Wed, 25 Jul 2012, 13:00 EDT	Mon, 30 Jul 2012, 15:00 EDT
Resolved	Software System	xray unavailable	xray	Fri, 20 Jul 2012, 08:00 EDT	Fri, 20 Jul 2012, 14:30 EDT	The login problem has been resolved.
Resolved	Hardware	Sierra's scratch dirs offline	storage at SDSC	Tue, 10 Jul 2012, 13:45 PDT		Scratch dirs are once again available. The faulted disk has been located and the spare is in it's place.
Resolved	Software System	Hotel VMMs need to be rebooted.	hotel	Tue, 10 Jul 2012, 14:56 EDT	Tue, 10 Jul 2012, 16:00 EDT
Resolved	Software System	eucalyptus outage (sierra) (resolved)	sierra	Mon, 09 Jul 2012, 15:30 EDT		sierra is back online.
Resolved	Maintenance	Monthly maintenance Tuesday, 3 July	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 03 Jul 2012, 08:00 EDT	Tue, 03 Jul 2012, 23:59 EDT	Eucalyptus needs some more works, but the rest of the systems are back to normal. Please report any issues by submitting a ticket.
Resolved	Maintenance Software System	Sierra Nimbus cloud outage	sierra	Fri, 22 Jun 2012, 05:00 EDT	Fri, 22 Jun 2012, 12:00 EDT	Cleaned up the resource database. Users may need to re-add their public keys to the ec2 query interface.
Resolved	Hardware	Hotel filesystem problems	hotel	Thu, 21 Jun 2012, 08:30 CDT	Fri, 06 Jul 2012, 17:00 CDT	The Nimbus cloud and GPFS on hotel are working.
Resolved	Network	Upcoming Networking Outage for Sierra	sierra storage at SDSC	Tue, 05 Jun 2012, 17:00 PDT	Tue, 05 Jun 2012, 19:00 PDT
Resolved	Software System	xray back online (was xray offline)	xray	Tue, 05 Jun 2012, 18:00 EDT	Wed, 06 Jun 2012, 15:00 EDT	Xray had a failed compute Module. After disabling that we were able to reboot the system.
Resolved	Hardware	foxtrot storage failure	foxtrot	Tue, 05 Jun 2012, 12:00 EDT		System board has been replaced (twice, as the first replacement board was also defective). There was no data loss.
Resolved	Software System	Inca outage	other	Tue, 05 Jun 2012, 12:00 EDT		We have restored the database from a May 1 backup and current data is once again being collected.
Resolved	Maintenance	Monthly maintenance Tuesday, 5 June	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 05 Jun 2012, 08:00 EDT	Tue, 05 Jun 2012, 23:59 EDT	Xray and foxtrot are still having issues, but rest of the systems are back to normal. Please report any issues by submitting a ticket.
Resolved	Software System	Inca outage	other	Tue, 05 Jun 2012, 08:00 EDT	Tue, 05 Jun 2012, 12:00 EDT	Maintenance has been completed.
Resolved	Power	sierra back online (was sierra offline, Thu 05/31)	sierra	Thu, 31 May 2012, 10:30 EDT	Thu, 31 May 2012, 15:00 EDT	The electrical work is finished. All services in sierra are back online.
Resolved	Software System	sierra file server reboot	sierra	Tue, 29 May 2012, 15:00 EDT	Tue, 29 May 2012, 15:45 EDT	sierra file server is back online.
Resolved	Hardware	sierra back online (was sierra unavailable (electrical issue))	sierra	Tue, 22 May 2012, 16:45 EDT	Tue, 15 May 2012, 22:00 EDT	All services have been restored. Sierra is back online.
Resolved	Hardware	sierra back online (was sierra unavailable (electrical issue))	sierra	Wed, 16 May 2012, 08:00 EDT	Wed, 16 May 2012, 13:00 EDT	Power has been restored. All the nodes in sierra have been rebooted and all the services are back online. Please submit a ticket if you notice any issues.
Resolved	Software System	Eucalyptus version 3.0.2 installation	india sierra	Mon, 14 May 2012, 08:00 EDT	Tue, 15 May 2012, 07:59 EDT	Eucalyptus 3.0.2 has been successfully installed in india and sierra. Accounts and credential files have been generated for all valid portal users. Look for credential files in your home directory under .futuregrid/eucalyptus folder. You will need to use euca2ools-2.0.2 with Eucalyptus 3 (available via module load euca2oos/2.0.2). Remember to create new keys and apply proper euca-authorize commands. Details tutorial is here: http://archive.futuregrid.org/tutorials/eucalyptus3.
Resolved	Software System	india down (resolved)	india	Sat, 12 May 2012, 17:56 EDT	Sun, 13 May 2012, 22:00 EDT	The nfs server was unresponsive. A reboot brought it back up.
Resolved	Software System	Quick change to hotel's Nimbus cloud	hotel	Fri, 11 May 2012, 18:50 EDT	Fri, 11 May 2012, 20:00 EDT
Resolved	Software System	india is not available	india	Fri, 11 May 2012, 13:40 EDT		The storage(NFS) server was being hang up. We restarted the server and reexport the nfs partition.
Resolved	Software System	Nimbus cleanup on hotel.	hotel	Thu, 10 May 2012, 10:31 HST	Thu, 10 May 2012, 16:00 HST
Resolved	Maintenance Hardware	Hotel home filesystem unavailable	hotel	Wed, 09 May 2012, 12:30 CDT	Thu, 10 May 2012, 08:00 CDT	The filesystem has been repaired and everything is back online.
Resolved	Maintenance	Portal maintenance 5/8	other	Tue, 08 May 2012, 08:00 CDT	Tue, 08 May 2012, 10:00 CDT	The updates and upgrades have been applied. If you notice any issues please file a ticket.
Resolved	Software System Hardware	Cloud maintenance	hotel	Tue, 08 May 2012, 07:15 EDT	Tue, 08 May 2012, 19:15 EDT
Resolved	Software System	Inca outage	other	Tue, 01 May 2012, 13:00 PDT	Tue, 01 May 2012, 17:34 PDT	Maintenance has been completed.
Resolved	Maintenance	Monthly maintenance Tuesday, 1 May	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 01 May 2012, 08:00 EDT	Tue, 01 May 2012, 23:59 EDT	All systems are back to normal. Please report any issues by submitting a ticket.
Resolved	Network	Network issue	alamo	Thu, 26 Apr 2012, 10:30 EDT	Fri, 27 Apr 2012, 12:00 EDT	Alamo network not affected by router issue.
Resolved	Software System	india unavailable	india	Tue, 24 Apr 2012, 20:50 EDT	Wed, 25 Apr 2012, 09:00 EDT	India's internal dhcp server was having a problem. We fixed it and india is back online now.
Resolved	Software System	Alamo Outage	alamo	Sat, 21 Apr 2012, 23:00 EDT	Sun, 22 Apr 2012, 15:00 EDT	The partition now has some free space so the cluster should be operating correctly again. If you saved a virtual machine to Nimbus recently and can no longer see it in your list of VMs, please submit a ticket. We have 2 VMs that were mis-filed and one of them may be yours.
Resolved	Software System	eucalyptus restart tonight (resolved)	india	Wed, 18 Apr 2012, 22:00 EDT	Wed, 18 Apr 2012, 23:00 EDT	A full restart of the eucalyptus cloud and cluster controller fixed the issue. All services are back to normal.
Resolved	Hardware	xray unavailable	xray	Wed, 18 Apr 2012, 13:30 EDT	Thu, 19 Apr 2012, 13:00 EDT	xray has been returned to service.
Resolved	Hardware	Hotel unavailable due to unplanned cooling failure	hotel	Sat, 14 Apr 2012, 22:00 CDT		All Hotel resources should now be available for use.
Resolved	Maintenance Software System	Nimbus Hotel down for cleanup	hotel	Wed, 11 Apr 2012 (All day)	Wed, 11 Apr 2012 (All day)
Resolved	Hardware	xray unavailable	xray	Wed, 04 Apr 2012, 15:20 EDT	Thu, 05 Apr 2012, 13:00 EDT	xray has been returned to service.
Resolved	Maintenance	Monthly maintenance Tuesday, 3 April	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 03 Apr 2012, 09:00 EDT	Tue, 03 Apr 2012, 11:59 EDT
Resolved	Software System	Nimbus Unavailable on Alamo	alamo	Sat, 31 Mar 2012, 13:30 EDT	Thu, 12 Apr 2012, 22:00 EDT	Nimbus is available on Alamo. We apologize for the extended downtime while we worked through a number of issues. Please submit a ticket if you encounter any problems. We upgraded the Nimbus nodes to CentOS 6 which allows us to use much newer versions QEMU, KVM, and libvirt. We upgraded to Nimbus 2.9 earlier during the downtime and we continue to use this version of Nimbus. These upgrades appear to have resolved the network crashes and kernel panics that were occurring under heavy Nimbus load.
Resolved	Software System	eucalyptus restart (resolved)	india	Fri, 16 Mar 2012, 13:00 EDT	Fri, 16 Mar 2012, 14:00 EDT	Clearing up some large instances and restarting the cluster fixed the issue. All services are back to normal. Please submit a ticket if you notice any issues.
Resolved	Software System	Alamo Nimbus partition outage	alamo	Fri, 16 Mar 2012, 07:30 EDT	Fri, 16 Mar 2012, 05:30 EDT	Nimbus has been upgraded to the latest version (2.9) on Alamo. We hope this will stop the network crashes - running the latest version will at least make it easier for us to debug if they reoccur.
Resolved	Software System Network	Alamo nimbus partition outage	alamo	Mon, 12 Mar 2012, 12:00 EDT	Thu, 15 Mar 2012, 17:30 EDT	Initially we retired an older switch that was throwing errors. We also had an issue with nimbus VM's not shutting down correctly. The worksp were being re-used, so duplicate mac addresses were being used on the new switch. Nimbus partition was put back in production last night.
Resolved	Maintenance	Portal outage for updates and upgrades	other	Thu, 08 Mar 2012, 15:00 CST	Thu, 08 Mar 2012, 15:30 CST	Maintenance is completed and the module updates/upgrades have been applied.
Resolved	Software System	sierra	sierra	Wed, 07 Mar 2012, 12:31 EST	Wed, 07 Mar 2012, 03:00 EST	2.9 is now installed
Resolved	Maintenance	Monthly maintenance Tuesday, 6 March	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 06 Mar 2012, 08:00 EST	Tue, 06 Mar 2012, 23:59 EST	All systems are back to normal. Please report any issues by submitting a ticket.
Resolved	Network	Alamo network outage	alamo	Sat, 03 Mar 2012, 10:30 EST	Tue, 06 Mar 2012, 17:00 EST	All network switches have been upgraded to latest firmware and an older switch has been retired. We have reconfigured the 10G uplink to it's original path. Both Nimbus and HPC partitions have been running without issue and network seems stable. Will continue to monitor for hardware errors.
Resolved	Network	Alamo Network outage	alamo	Tue, 28 Feb 2012, 01:20 EST	Fri, 02 Mar 2012, 12:00 EST	Problematic switch identified and marked for replacement. It may be the Friday before I get a replacement switch. Alamo cluster is up and running but external interfaces on nimbus partition are not working. NIMBUS partition still unavailable. Removing the offending switch and re-installing the nimbus partition seems to have fixed the problem with the network crashing. Will continue to monitor closely. Nimbus partition is available again.
Resolved	Network	Alamo network outage	alamo	Sat, 25 Feb 2012, 08:00 EST	Sat, 25 Feb 2012, 12:00 EST	An uplink switch was rebooted and connectivity was restored.
Resolved	Network	intermittent access to Alamo this weekend	alamo	Sat, 18 Feb 2012, 08:00 CST	Mon, 20 Feb 2012, 17:00 CST	DNS server referenced in VM's configuration has been restored. Test nimbus sessions show external connectivity now.
Resolved	Network	Inca and Netlogger outage	other	Tue, 07 Feb 2012, 08:00 PST		This outage was resolved by Feb 10. An older version of VM server where these services are hosted was restored so monitoring data from Jan 18 - Feb 8 was lost. Status pages linked from http://archive.futuregrid.org/status are available again.
Resolved	Maintenance	Monthly maintenance Tuesday, 7 February	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 07 Feb 2012, 08:00 EST	Tue, 07 Feb 2012, 23:59 EST	All FutureGrid systems have been returned to service. Please send email to help@futuregrid.org if you notice any issues.
Resolved	Maintenance	Portal maintenance/updates	other	Thu, 02 Feb 2012, 22:00 CST	Thu, 02 Feb 2012, 22:30 CST	Portal updates have been applied and maintenance is complete. Please report any problems or issues with the portal by submitting a ticket at http://archive.futuregrid.org/help.
Resolved	Maintenance	Hotel unavailable for filesystem rebuilds	hotel	Tue, 24 Jan 2012, 10:45 CST	Sat, 28 Jan 2012, 20:30 CST	All filesystems have been rebuilt and all data restored. Both the batch and nimbus nodes have been released for general use. Please report any issues to help@futuregrid.org.
Resolved	Maintenance	Inca and Netlogger outage this afternoon	other	Wed, 18 Jan 2012, 15:00 EST	Wed, 18 Jan 2012, 17:00 EST	Maintenance has been completed.
Resolved	Software System	india offline	india	Thu, 12 Jan 2012, 15:30 EST		india has been returned to service. Please send email to help@futuregrid.org if you notice any problems.
Resolved	Software System Hardware	Hotel /home and /software filesystem outage	hotel	Thu, 05 Jan 2012, 05:30 CST	Thu, 05 Jan 2012, 09:00 CST	The affected servers have been rebooted and filesystems remounted and nimbus has been restarted.
Resolved	Software System	india down	india	Wed, 04 Jan 2012 (All day)		Fixed the problem on the communication between NFS and compute nodes.
Resolved	Maintenance	Monthly Maintenance 3 January	hotel	Tue, 03 Jan 2012, 08:00 EST	Tue, 03 Jan 2012, 15:00 CST	Monthly maintenance of hotel has completed and all services should functional.
Resolved	Maintenance Hardware	Hotel GPFS scratch space being rebuilt	hotel	Tue, 03 Jan 2012, 07:00 CST	Tue, 03 Jan 2012, 15:00 CST	The GPFS scratch filesystem has been rebuilt with considerable performance increases: almost 7.5x better write performance: 550 MB/s as opposed to the 75 MB/s before almost 4x better read performance: 900 MB/s as opposed to the 230 MB/s before During the month of January, we will be syncing over the home, software, and images GPFS filesystems to scratch so that they can be rebuilt during the February maintenance so performance may be impacted during the night from midnight to 6am.
Resolved	Maintenance	Alamo chilled water outage	alamo	Tue, 27 Dec 2011, 06:00 EST	Fri, 30 Dec 2011, 17:00 EST	Alamo in production
Resolved	Hardware	foxtrot is unreacheable due to network switch failure	foxtrot	Mon, 19 Dec 2011, 12:00 EST	Mon, 26 Dec 2011, 12:00 EST	Switch replacement has been received from Force10, and its configuration has been restored.
Resolved	Power	foxtrot unavailable 12/19/2011 (Mon)	foxtrot	Mon, 19 Dec 2011, 07:00 EST		Power has been restored.
Resolved	Software System	jira service may become unavailable	other	Tue, 13 Dec 2011, 20:00 EST	Wed, 14 Dec 2011, 17:00 EST
Resolved	Maintenance	India outage	india	Wed, 07 Dec 2011 (All day)	Thu, 15 Dec 2011, 09:00 EST	The india cluster has been returned to service. Please email help@futuregrid.org if you notice any problems.
Resolved	Maintenance	FutureGrid Monthly Maintenance 6 December	foxtrot sierra xray	Tue, 06 Dec 2011, 08:00 EST	Tue, 06 Dec 2011, 23:59 EST	Maintenance on all FutureGrid systems has been completed, and all systems have been returned to service with the exception of the india cluster. The india cluster will be offline through 14 December for software scalability testing. Please email help@futuregrid.org for any problems.
Resolved	Software System	sierra offline	sierra	Sat, 26 Nov 2011, 11:29 EST		sierra has been returned to service. Please email help@futuregrid.org with any issues.
Resolved	Software System	Eucalyptus unavailable for users	india sierra	Tue, 08 Nov 2011 (All day)		The xen network bridge (xenbr0) was not attaching the network interface (eth0) properly. Even though the instances were running, we weren't able to connect to them. This started after we applied the server wide redhat update. Removing xen, reinstalling and rebooting the nodes fixed the issue. All services are running now. Please report if you notice any issues.
Resolved	Maintenance Power	Hotel unavailable for planned power outage and monthly maintenance	hotel	Fri, 04 Nov 2011, 15:00 CDT	Wed, 09 Nov 2011, 16:00 CST	Maintenance on Hotel has completed. All services should be available. Please note that new, stricter firewall rules are now in place on all machines so if you have any issues, please email help@futuregrid.org.
Resolved	Maintenance	Monthly maintenace Tuesday, 1 November	india foxtrot sierra xray alamo storage at SDSC storage at IU network	Tue, 01 Nov 2011, 08:00 EDT	Tue, 01 Nov 2011, 23:59 EDT	All FutureGrid systems have been returned to service, with the exception of Eucalyptus on india and sierra. System engineers are still working on the Eucalyptus instances, and they are expected to be returned to service shortly.
Resolved	Software System	xray down	xray	Thu, 20 Oct 2011, 23:25 EDT	Fri, 21 Oct 2011, 01:30 EDT	xray has been returned to service.
Resolved	Network	SDSC Network Maintenance	sierra storage at SDSC	Tue, 18 Oct 2011, 19:00 PDT	Tue, 18 Oct 2011, 20:00 PDT	Work was completed last night.
Resolved	Maintenance	FutureGrid maintenance 4 October 2011	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 04 Oct 2011, 07:00 EDT	Tue, 04 Oct 2011, 23:59 EDT	Monthly maintenance on FutureGrid resources has been completed and the resources have been returned to service. Please send email to help@futuregrid.org for any issues.
Resolved	Software System	Hotel GPFS Outage	hotel	Wed, 28 Sep 2011, 17:00 CDT	Wed, 28 Sep 2011, 22:35 CDT
Resolved	Maintenance	Portal Maintenance 9/26	other	Mon, 26 Sep 2011, 22:30 CDT	Mon, 26 Sep 2011, 11:30 CDT	Portal maintenance has been completed.
Resolved	Software System	Eucalyptus Outage	india	Mon, 26 Sep 2011, 20:27 EDT		Eucalyptus is back online. We are investigating the details of this outage. At this point we are not entirely sure what caused it but error logs seem to point to file permission issues.
Resolved	Software System	Inca outage	other	Mon, 26 Sep 2011, 12:00 PDT		Service has been restored.
Resolved	Network	sierra offline	sierra storage at SDSC	Tue, 20 Sep 2011, 18:12 PDT	Tue, 20 Sep 2011, 21:00 EDT	All SDSC systems have been returned to service after a planned network upgrade.
Resolved	Network	SDSC networking upgrade	sierra storage at SDSC	Tue, 20 Sep 2011, 17:00 PDT	Tue, 20 Sep 2011, 21:00 PDT	Network upgrades have been completed.
Resolved	Maintenance Network	Network maintenance at IU, Tuesday 20 September	india xray storage at IU network	Tue, 20 Sep 2011, 14:00 EDT	Tue, 20 Sep 2011, 15:00 EDT	Network maintenance was completed on schedule.
Resolved	Software System	Eucalyptus Outage on India	india	Thu, 15 Sep 2011, 14:30 EDT	Thu, 08 Sep 2011, 16:30 EDT	We couldn't figure out the issue. However for we changed the network mode to novlan which got us up and running.
Resolved	Software System	leggett	hotel	Wed, 14 Sep 2011, 16:03 CDT	Wed, 14 Sep 2011, 23:03 CDT
Resolved	Software System Network	Eucalyptus Outage on India	india	Mon, 12 Sep 2011, 17:00 EDT	Mon, 12 Sep 2011, 20:00 EDT
Resolved	Power	San Diego power outage	sierra storage at SDSC	Thu, 08 Sep 2011, 15:19 PDT		All the services are back up and operating. Report to help@futuregrid.org if you notice any issues.
Resolved	Maintenance	leggett	hotel	Tue, 06 Sep 2011, 07:00 CDT	Thu, 15 Sep 2011, 11:19 CDT
Resolved	Maintenance	Monthly maintenance 6 September	india foxtrot sierra hotel xray alamo storage at SDSC storage at IU network	Tue, 06 Sep 2011, 07:00 EDT	Tue, 06 Sep 2011, 23:59 EDT	All FutureGrid systems have been returned to service. Please email help@tickets.futuregrid.org with any issues.
Resolved	Software System	Nimbus failures on Alamo	alamo	Fri, 02 Sep 2011, 12:00 EDT	Fri, 02 Sep 2011, 14:30 EDT	The Nimbus configuration was corrected to store virtual machine images on the intended (large) disk partition.
Resolved	Software System	Sierra Outage for Nimbus 2/8 upgrade	sierra	Thu, 25 Aug 2011, 17:00 EDT	Fri, 26 Aug 2011 (All day)
Resolved	Power	foxtrot unavailable for power outage	foxtrot	Sat, 13 Aug 2011, 06:00 EDT
Resolved	Power	sierra offline	sierra	Fri, 05 Aug 2011, 16:57 PDT	Sat, 06 Aug 2011, 09:00 PDT	sierra has been returned to service. Please email help@futuregrid.org if you encounter any problems.
Resolved	Power	Sierra unavailable due to UCSD power outage	sierra storage at SDSC	Fri, 05 Aug 2011, 04:40 PDT	Sat, 06 Aug 2011, 09:00 EDT	sierra has been returned to service. Please email help@futuregrid.org if you encounter any problems.
Resolved	Hardware	Hotel software filesystem down	hotel	Thu, 04 Aug 2011, 09:00 CDT	Thu, 04 Aug 2011, 17:00 CDT	The filesystem has been brought back online and should now be available.
Resolved	Maintenance	Hotel Monthly Maintenance	hotel	Tue, 02 Aug 2011, 07:00 CDT	Wed, 03 Aug 2011, 17:00 CDT	The monthly maintenance has been completed.
Resolved	Maintenance	Monthly FutureGrid maintenance	india foxtrot sierra xray storage at SDSC storage at IU	Tue, 02 Aug 2011, 08:00 EDT	Tue, 02 Aug 2011, 23:59 EDT	Maintenance has been completed on india, foxtrot, sierra, and xray and the systems have been returned to service. Please send email to help@tickets.futuregrid.org for any issues.
Resolved	Maintenance	alamo monthly maintenance Extended	alamo	Tue, 02 Aug 2011, 06:21 CDT	Wed, 03 Aug 2011, 12:22 EDT	Problem with OFED stack took a little longer to debug than I hoped. Alamo back in production.
Resolved	Network	Alamo network connectivity	alamo	Mon, 01 Aug 2011, 17:00 EDT		Temporary DNS name search issue has been resolved on nimbus VM's.
Resolved	Hardware	Emergency shutdown of hotel due to cooling loss	hotel	Mon, 25 Jul 2011, 06:00 CDT	Mon, 25 Jul 2011, 14:30 CDT	Hotel has been returned to service and jobs that were queued previous to the shutdown have begun running. Jobs that were running when we shutdown were killed probably without any notification going out.
Resolved	Network Power	foxtrot maintenance	foxtrot	Fri, 22 Jul 2011, 06:00 EDT	Sat, 23 Jul 2011, 06:00 EDT
Resolved	Software System	Eucalyptus Outage on India	india	Thu, 14 Jul 2011, 20:30 EDT	Thu, 14 Jul 2011, 21:00 EDT
Resolved	Maintenance Power	Hotel unavailable for planned power outage and monthly maintenance	hotel	Tue, 12 Jul 2011, 09:00 CDT	Tue, 12 Jul 2011, 22:00 CDT	Power work was completed and all maintenance concluded.
Resolved	Maintenance	Alamo nimbus partition unavailable	alamo	Mon, 11 Jul 2011, 09:00 CDT	Mon, 11 Jul 2011, 17:00 CDT	After lengthy fsck, the /N/images file system is on line. The Nimbus partition is available again.
Resolved	Network	Hotel network outage	hotel	Thu, 07 Jul 2011 (All day)	Thu, 07 Jul 2011, 06:00 CDT
Resolved	Power	Hotel unavailable for unplanned power outage	hotel	Wed, 06 Jul 2011, 08:30 CDT	Wed, 06 Jul 2011, 13:30 CDT	Power was restored and the machine brought back online.
Resolved	Maintenance	FutureGrid monthly maintenance 7 July	india foxtrot xray storage at IU	Tue, 05 Jul 2011, 08:00 EDT	Tue, 05 Jul 2011, 23:59 EDT	Maintenance has been completed for india, xray, and foxtrot, and the systems have been returned to production. Please send email to help@tickets.futuregrid.org for any issues. The sierra cluster maintenance has been extended, but is expected to be returned to service tomorrow.
Resolved	Software System	Alamo filesystem issues	alamo	Tue, 28 Jun 2011, 15:30 CDT	Wed, 29 Jun 2011, 12:30 CDT	File system: /N/images has been restored from backups. nimbus partition is now available. File system: /N/work temporary file system was purged.
Resolved	Network	Alamo emergency network outage	alamo	Mon, 27 Jun 2011, 17:00 CDT	Mon, 27 Jun 2011, 18:00 CDT	Network maintenance to fiber completed. Connection appears to be reliable.
Resolved	Hardware	xray offline for hardware diagnostics	xray	Fri, 24 Jun 2011, 08:09 EDT	Mon, 27 Jun 2011, 19:38 EDT
Resolved	Network	FutureGrid Participant SDSC Maintenance Update	sierra network	Thu, 23 Jun 2011, 09:00 UTC	Thu, 23 Jun 2011, 12:00 UTC	Note this maintenance has been Cancelled
Resolved	Maintenance Network	Intermittent access to Alamo	alamo	Tue, 21 Jun 2011, 09:00 CDT	Tue, 21 Jun 2011, 12:00 CDT	Maintenance is complete.
Resolved	Network	india cluster reboot	india	Tue, 14 Jun 2011, 21:14 EDT	Tue, 14 Jun 2011, 23:15 EDT	System returned to service.
Resolved	Network	Qwest fiber splicing affecting Hotel network	hotel	Fri, 10 Jun 2011, 23:00 CDT	Sat, 11 Jun 2011, 05:00 CDT	This activity has been postponed for an unknown date.
Resolved	Network	Qwest fiber splicing affecting Hotel network	hotel	Thu, 09 Jun 2011, 23:00 CDT	Fri, 10 Jun 2011, 05:00 CDT	This activity has been postponed for an unknown date.
Resolved	Maintenance	Monthly system maintenance	india foxtrot sierra xray network	Tue, 07 Jun 2011, 08:00 EDT	Tue, 07 Jun 2011, 23:59 EDT	The india, sierra, foxtrot, and xray resources have all been returned to production. Please email help@tickets.futuregrid.org if you notice any problems.
Resolved	Hardware	xray unavailable	xray	Mon, 06 Jun 2011, 08:00 EDT	Mon, 06 Jun 2011, 10:30 EDT	xray has been returned to service.
Resolved	Software System	Eucalyptus on India	india	Fri, 03 Jun 2011, 10:25 EDT	Sat, 04 Jun 2011, 12:00 EDT	Eucalyptus on India has been restored. The instances have lost their public IPs, so users will need to reallocate IPs to the instances.
Resolved	Software System	Alamo Nimbus DNS problem	alamo	Wed, 01 Jun 2011, 16:00 EDT	Thu, 02 Jun 2011, 17:30 EDT	Default name servers changed to be ones that are accessible under the current network configuration.
Resolved	Power	Hotel unavailable for planned power outage	hotel	Fri, 27 May 2011, 15:00 CDT	Thu, 02 Jun 2011, 23:00 CDT	Maintenance has concluded and all systems have been released back for work.
Resolved	Software System	alamo maintenance extend to 5:00 PM	alamo	Fri, 27 May 2011, 12:00 CDT	Fri, 27 May 2011, 17:00 CDT	NFS mounts moved from IB interface to Gig E interface on compute nodes for stability.
Resolved	Network	Alamo login node unavailable	alamo	Thu, 26 May 2011, 17:30 CDT	Fri, 27 May 2011, 17:00 EDT
Resolved	Network	alamo maintenance complete	alamo	Thu, 26 May 2011, 17:27 EDT	Fri, 27 May 2011, 17:00 EDT	Compute nodes were configured to mount NFS via the GigE interfaces. A config change was also needed for OFED stack.
Resolved	Hardware	xray down	xray	Tue, 24 May 2011, 06:29 EDT	Tue, 24 May 2011, 13:00 EDT	xray has been returned to service. Please send email to help@futuregrid.org if you envounter problems.
Resolved	Hardware	Hotel home filesystem down	hotel	Mon, 23 May 2011, 17:00 CDT	Tue, 24 May 2011, 23:00 CDT	All filesystems are back online and all services are back as well.
Resolved	Maintenance	india network maintenance Sunday, 22 May	india	Sun, 22 May 2011, 10:00 EDT	Sun, 22 May 2011, 12:00 EDT	india has been returned to service
Resolved	Software System	Nimbus unavailable on sierra	sierra	Wed, 11 May 2011, 10:00 EDT		Explanation can be found here: http://archive.futuregrid.org/revocation-policies-and-recent-sierra-cloud-outages
Resolved	Network	Alamo Network connectivity	alamo	Fri, 06 May 2011, 16:32 EDT	Tue, 24 May 2011, 11:30 EDT	The issue was resolved by reconfiguring the route through a different switch.
Resolved	Maintenance	Extended maintenance for foxtrot	foxtrot	Tue, 03 May 2011, 08:00 EDT	Wed, 04 May 2011, 12:00 EDT	System returned to production.
Resolved	Maintenance	Monthly maintenance Tuesday, 3 May	india foxtrot sierra xray	Tue, 03 May 2011, 08:00 EDT	Tue, 03 May 2011, 23:59 EDT
Resolved	Maintenance	Hotel Monthly Maintenance, Tuesday, May 3rd	hotel	Tue, 03 May 2011, 07:00 CDT	Wed, 04 May 2011, 23:59 CDT	Systems have been returned to service
Resolved	Maintenance	india, sierra, xray returned to service	india sierra xray	Tue, 03 May 2011, 08:00 EDT	Tue, 03 May 2011, 23:30 EDT	Systems have been returned to service
Resolved	Maintenance Software System	Eucalyptus on India outage	india	Mon, 02 May 2011, 12:00 EDT	Sat, 07 May 2011, 12:00 EDT	Eucalyptus has been restored to service and now has 252 IPs that users can assign to VMs split into 2 pools for the india and india26-50 zones. Thank you for your patience.
Resolved	Maintenance	Resource reallocation	india	Mon, 25 Apr 2011, 11:45 EDT	Mon, 25 Apr 2011, 12:00 EDT	Eucalyptus is back in production. Some VMs dont have public IP assignments and can be reached from i135 over the private network.
Resolved	Network	SDSC network outage	sierra storage at SDSC	Thu, 21 Apr 2011, 06:00 UTC	Thu, 21 Apr 2011, 08:00 UTC	Participant SDSC remained available while their circuit vendor performed maintenance in Grafton, NE. No outages occurred. Maintenance has been completed.
Resolved	Hardware	Sierra HPC Outage	sierra storage at SDSC	Fri, 15 Apr 2011, 13:16 EDT
Resolved	Maintenance	Monthly maintenance 5 April 2011	india foxtrot sierra xray	Tue, 05 Apr 2011, 08:00 EDT	Tue, 05 Apr 2011, 23:59 EDT
Resolved	Maintenance	Monthly maintenance 5 April 2011	hotel	Tue, 05 Apr 2011, 07:00 CDT	Tue, 05 Apr 2011, 23:00 CDT	The monthly maintenance of hotel has completed.
Resolved	Network	SDSC network outage	sierra	Thu, 31 Mar 2011, 06:00 UTC	Thu, 31 Mar 2011, 11:00 UTC	Outage was cancelled and will be rescheduled at a later date yet to be determined.
Resolved	Maintenance	India Eucalyptus Outage	india	Wed, 23 Mar 2011, 16:55 EDT	Wed, 23 Mar 2011, 17:00 EDT	The CC was restarted.
Resolved	Network	SDSC network outage	sierra	Tue, 15 Mar 2011, 17:00 PDT	Tue, 15 Mar 2011, 21:00 PDT	Completed.
Resolved	Network	india back online after brief outage	india	Thu, 10 Mar 2011, 11:41 EST	Thu, 10 Mar 2011, 13:41 EST	The india cluster is available for users.
Resolved	Hardware	foxtrot storage system upgrade	foxtrot	Thu, 10 Mar 2011, 08:00 EST	Thu, 10 Mar 2011, 13:00 EST
Resolved	Maintenance	sierra maintenance 1 March 2011	sierra	Tue, 01 Mar 2011, 08:00 PST	Tue, 01 Mar 2011, 23:59 PST	sierra has been returned to service.
Resolved	Maintenance	Eucalytpus on Sierra and India	india sierra	Tue, 01 Mar 2011, 09:00 EST	Tue, 01 Mar 2011, 17:00 EST	Eucalyptus has been returned to service.
Resolved	Maintenance	hotel monthly maintenance	hotel	Tue, 01 Mar 2011, 07:00 CST	Wed, 02 Mar 2011, 17:00 CST	Maintenance has concluded. NOTE: There are now soft and hard quotas for all user's home filesystem. The soft quota is 10GB with a grace period of 7 days, and the hard quota is 20 GB. If you don't get below your soft quota before the end of the grace period or you hit your hard quota, you will not be able to add any more data to your home filesystem.
Resolved	Maintenance	india maintenance 1 March 2011	india	Tue, 01 Mar 2011, 08:00 EST	Tue, 01 Mar 2011, 23:59 EST	india has been returned to service.
Resolved	Maintenance	xray maintenance 1 March 2011	xray	Tue, 01 Mar 2011, 08:00 EST	Tue, 01 Mar 2011, 23:59 EST	xray has been returned to service.
Resolved	Maintenance	foxtrot maintenance 1 March 2011	foxtrot	Tue, 01 Mar 2011, 08:00 EST	Tue, 01 Mar 2011, 23:59 EST	foxtrot has been returned to service
Resolved	Software System	xray available	xray	Sat, 26 Feb 2011, 18:16 EST	Sun, 27 Feb 2011, 09:00 EST
Resolved	Software System	xray unavailable	xray	Fri, 25 Feb 2011, 18:16 EST	Sat, 26 Feb 2011, 12:00 EST
Resolved	Software System	xray returned to service	xray	Sun, 06 Feb 2011, 11:42 EST	Sun, 06 Feb 2011, 13:45 EST	System was rebooted.
Resolved	Maintenance	sierra monthly maintenance	sierra	Tue, 01 Feb 2011, 05:00 PST	Wed, 02 Feb 2011, 05:00 PST
Resolved	Maintenance	foxtrot monthly maintenance	foxtrot	Tue, 01 Feb 2011, 08:00 EST	Wed, 02 Feb 2011, 08:00 EST
Resolved	Maintenance Network	FutureGrid network maintenance at IU	india xray storage at IU network	Tue, 01 Feb 2011, 08:00 EST	Tue, 01 Feb 2011, 09:30 EST
Resolved	Maintenance	Hotel monthly maintenance	hotel	Tue, 01 Feb 2011, 07:00 CST	Tue, 01 Feb 2011, 17:00 CST
Resolved	Maintenance	india monthly maintenance	india	Tue, 01 Feb 2011, 08:00 EST	Wed, 02 Feb 2011, 08:00 EST
Resolved	Maintenance	xray monthly maintenance	xray	Tue, 01 Feb 2011, 08:00 EST	Wed, 02 Feb 2011, 08:00 EST
Resolved	Hardware	Sierra storage node outage	sierra storage at SDSC	Fri, 28 Jan 2011, 10:00 PST	Fri, 28 Jan 2011, 11:26 PST	Sierra is back up and available to users.
Resolved	Hardware	Sierra storage node outage	sierra storage at SDSC	Tue, 25 Jan 2011, 10:00 PST	Tue, 25 Jan 2011, 10:52 PST	Sierra is back up and available to users.
Resolved	Network	Hotel storage node outage	hotel	Tue, 18 Jan 2011, 01:21 CST	Tue, 18 Jan 2011, 08:30 CST
Resolved	Maintenance	Hotel monthly maintenance	hotel	Tue, 11 Jan 2011, 07:00 CST	Wed, 12 Jan 2011, 12:00 CST
Resolved	Maintenance	Chilled water outage	alamo	Mon, 03 Jan 2011, 06:00 EST	Wed, 05 Jan 2011, 17:00 EST	Alamo computes nodes are on line and queues have been re-enabled.

FutureGrid Outage Archive

About

Support

Community

Projects