Phoenix Cluster Migration to Slurm Scheduler

Dear Phoenix researchers,

The Phoenix cluster will be migrating to the Slurm scheduler over the next couple of months with the first phase scheduled for October 10! PACE has worked closely with the PACE Advisory Committee (PAC) on a plan for the migration to ensure minimum interruption to research. Slurm is a widely popular, open-source scheduler on many research computing clusters, so you may have experienced it elsewhere. If commands like ‘sbatch’ and ‘squeue’ sound familiar to you, then you have used Slurm! Phoenix will be the second cluster (after Hive) in PACE’s transition from Torque/Moab to Slurm. We expect the new scheduler to provide improved stability and reliability, offering a better user experience. We will be updating our software stack at the same time and offering support with orientation and consulting sessions to facilitate this migration.

The phased transition is planned in collaboration with the faculty-led PACE Advisory Committee which is comprised of a representative group of PACE and faculty members. We are planning a staggered phased migration for the Phoenix cluster. The six phases include the following dates and number of nodes:

The first phase will begin October 10, during which 500 Phoenix compute nodes (of 1319 total) will join our new “Phoenix-Slurm” cluster while the rest will remain on the existing Phoenix cluster. The 500 nodes will represent each existing node type proportionally. Following the first phase, we strongly encourage all researchers to begin shifting over their workflows to the Slurm-based side of Phoenix to take advantage of the improved features and queue wait times. Also, as part of the phased migration approach, researchers will continue to have access to the existing Phoenix cluster that will last until the final phase of this migration to ensure minimum interruption to research. Users will receive detailed communication on how to connect to the Phoenix-Slurm cluster along with other documentation and training.

In addition to the scheduler migration, another significant change for researchers on Phoenix will be an update to the PACE Apps software stack. The Phoenix-Slurm cluster will feature a new set of provided applications listed in our documentation. Please review this list of software we plan to offer on Phoenix post-migration and let us know via email (pace-support@oit.gatech.edu) if any software you are currently using on Phoenix is missing from that list. We encourage you to let us know as soon as possible to avoid any potential delay to your research as the migration process concludes. We have reviewed batch job logs to determine packages in use and upgraded them to the latest version. Researchers installing or writing their own software will also need to recompile applications to reflect new MPI and other libraries.

Starting after the November PACE Maintenance period (November 2), we will no longer be accepting software installation requests for new software on the existing Phoenix cluster with Torque/Moab. All software requests after November 2 will be for Phoenix-Slurm. Additionally, all new researcher groups joining PACE after November 2 will be onboarded onto Phoenix-Slurm only.

You will notice a few other changes to Phoenix in the new environment. As with the current Phoenix cluster, faculty and their research teams will receive the full free tier monthly allocation, equivalent to 10,000 CPU*hours on base hardware and usable on all architectures, on Phoenix-Slurm (in addition to the one on the existing Phoenix cluster) as well as access to Embers, our free backfill queue. We will be charging users for jobs on Phoenix-Slurm.

For prepaid accounts (including CODA20 refresh accounts), PACE will split your account balances 50/50 on the Phoenix-Slurm and existing Phoenix (with Torque/Moab) clusters during the migration. For new computing credits purchased after Nov 1st, 75% will be allocated to the Phoenix-Slurm cluster. For new computing credits purchased after Jan 3, 100% will be allocated to the Phoenix-Slurm cluster.

For postpaid (monthly) accounts, PACE will set the same limit based on existing credits on Phoenix to Phoenix-Slurm. Please be aware that for postpaid accounts this could lead to potential monthly overcharges if users were to run on both clusters to 100%. However, we wanted to allow researchers to have access to their full monthly limit for flexibility. For postpaid accounts, Principal Investigators and users are responsible for tracking their spending limit on the Phoenix-Slurm and Phoenix clusters to avoid going over budget.

PACE will provide documentation, training sessions, and additional support (e.g., increased frequency of PACE consulting sessions) to aid you as you transition your workflows to Slurm. Prior to the launch, we will have updated documentation as well as a guide for converting job scripts from PBS to Slurm-based commands. We will also offer specialized training virtual sessions (PACE Slurm Orientation) on the use of Slurm on Phoenix. Additionally, we have increased the frequency of our PACE consulting sessions during this migration phase for the Fall and Spring semesters, and you are invited to join our PACE Consulting Sessions or to email us for support. The schedule for the upcoming PACE Phoenix-Slurm orientation sessions will be provided in future communications.

We are excited to launch Slurm on Phoenix as we continue to improve Georgia Tech’s research computing infrastructure, and we will be providing additional information and support in the coming weeks through documentation, support tickets, and live sessions. Please contact us with any questions or concerns about this transition.

Best,

-The PACE Team