[Update 09/07/2022 10:23 AM]
Cooling has been fully restored to the datacenter, and operations have resumed on all PACE clusters, including Phoenix, Hive, Firebird, Buzzard, PACE-ICE, and COC-ICE. Jobs are now running, and new jobs may be submitted via the command line or Open OnDemand.
Any jobs that were running at the time of the outage have been cancelled and must be resubmitted by the researcher.
Refunds will be issued for jobs that were cancelled on clusters with charge accounts (Phoenix and Firebird).
Thank you for your patience as emergency repairs were completed. Please contact us at pace-support@oit.gatech.edu with any questions.
[Update 09/06/2022 4:51 PM]
Summary: Unfortunately, the same issue that happened yesterday with the primary cooling loop happened today with the secondary cooling loop. OIT operations and Databank requested that we power off all the compute nodes to repair the secondary cooling loop. We have captured all job information for Phoenix, Hive, COC-ICE, PACE-ICE, and Firebird and we will refund all running jobs on Firebird and Phoenix.
Impact: Researchers won’t be able to submit new jobs to the clusters either from the command line or from Open Ondemand webserver, the existing running jobs will be killed, and computing credits will be refunded if applied. Your partially ran jobs will lose their calculation if checkpointing is not supported in the job.
Sorry for the inconvenience and thank you for your patience; please contact us at pace-support@oit.gatech.edu with any questions. We will keep you posted with any updates.
[Update 09/06/2022 11:10 AM]
Cooling has been restored to the datacenter, and operations have resumed on all PACE clusters, including Phoenix, Hive, Firebird, Buzzard, PACE-ICE, and COC-ICE. Jobs are now running, and new jobs may be submitted via the command line or Open OnDemand. Any jobs that were running at the time of the outage have been cancelled and must be resubmitted by the researcher. Refunds will be issued for jobs that were cancelled on clusters with charge accounts (Phoenix and Firebird).
Thank you for your patience as emergency repairs were completed to avoid damage to the datacenter. Please contact us at pace-support@oit.gatech.edu with any questions.
[Update 09/06/2022 9:58 AM]
Around 8:00PM on 09/05/2022, OIT operations and Databank requested that PACE powers off all the compute nodes to avoid additional issues. PACE had captured all job information for Phoenix, Hive, COC-ICE, PACE-ICE, and Firebird and we will refund all running jobs on Firebird and Phoenix after they have been brought back online.
Currently, the cooling has been restored in Coda datacenter, and as of about 7:00am (09/06/22), PACE has been cleared to online and test the clusters before releasing them to users.
[Update 09/05/2022 8:45 PM]
[Original post]
Summary: One of the cooling towers in the CODA data center has issues, and the temperature is rising. We need to pause all PACE cluster schedulers, and possibly power down all compute nodes.
Impact: Researchers won’t be able to submit new jobs to the clusters either from the command line or the Open OnDemand web server; the existing jobs should continue running.
Thank you for your patience this afternoon; please contact us at pace-support@oit.gatech.edu with any questions. We will keep you posted with any updates.