Participation

Some critical questions from our faculty include:


"How can I participate?"

There are several straightforward options to consider:

1.  Immediate access to the shared resources available in the FoRCE Research Computing Environment.

This option provides faculty immediate access to the shared resources of the FoRCE cluster including compute nodes, GPUs, and basic storage.  Policies affecting the FoRCE cluster are maintained by the Faculty Governance Committee, and detailed on the policy page.

Cost: This option is provided at no additional cost to participants.

How to sign-up: Submit a brief proposal online here. (Note: You must be logged into the site using you GT credentials. Login here.)

2.  Faculty contribute nodes for shared access, augmenting the FoRCE cluster

Faculty are encouraged to use their HPC and research computing dollars to add nodes to the FoRCE.  In all cases, priority is given to the owner of these nodes.  When unused by the owner, these nodes may be utilized by other faculty who have also contributed nodes.  Policies affecting sharing are maintained by the Faculty Governance Committee, and detailed on the policy page.  By participating in this way, faculty will also have access to the FoRCE cluster and other shared clusters as time is available.  This option is good for faculty who must periodically have nodes reserved for their use, but also have workloads that can be handled by a shared queue. As another advantage, participants gain access to recent architectures in longer term as the FoRCE cluster grows in size with the addition of new nodes.  

Cost: As jobs in the shared environment may execute on other nodes, a baseline hardware configuration must be maintained.  Participants pay for the compute nodes they add to FoRCE as well as expanded storage.  (See #4)

 How to sign-up: Contact pace-support@oit.gatech.edu

3.  Faculty purchase compute nodes and expanded storage dedicated for exclusive use.

Faculty who require a cluster environment with dedicated compute nodes can purchase these nodes and still take advantage of the federated infrastructure.  These nodes are not shared with the FoRCE cluster or other shared clusters, and are available exclusively to the participant and to researchers they authorize for access. This option is good for faculty who expect to keep their nodes busy most of the time.

Cost: Participants pay for the compute nodes precisely sized to their requirements as well as expanded storage.  (See #4)

How to sign-up: Contact pace-support@oit.gatech.edu

4.  Faculty purchase expanded storage.

All user accounts are granted a 5GB home directory quota.  For long term storage of data sets, faculty may purchase dedicated storage to augment the existing base storage allocation by adding disk space to a project directory.  This storage is fully backed up and implemented using best-practice redundant disk arrays to provide data integrity and availability.  This option can be used independently of computational node model above.

Cost: Project storage is provided as a dedicated portion of a shared highly expandable DDN/GPFS storage system.  Storage may be purchased in increments as small as 1TB and may be easily expanded on demand.

How to sign-up: Contact pace-support@oit.gatech.edu

5. Faculty who want central hosting of a stand-alone non-federated cluster.

To maximize the value of every dollar invested in HPC, we strongly encourage participation in the federated cluster model. An existing cluster which simply needs floor space, power, cooling, and a network connection may be able to be hosted under the PACE Federation.  The impact of all such stand-alone requests will be evaluated on a case-by-case basis to ascertain the impact on the long-term availability of hosting facilities and associated resources.

Cost: TBD case-by-case.

How to sign-up: Contact pace-support@oit.gatech.edu to refine technical details, costs and options.

[ Return to top ]


"How much does it cost?"

PACE offers various compute options according to the table below.  We do not currently support the acquisition of compute elements based on AMD processors.  This pricing may vary over time as market conditions fluctuate. PACE does not levy any additional charges past the equipment costs shown below, as equipment costs are passed directly from vendors to faculty without markup.  Specific pricing will be provided at time of purchase. Participants are encouraged to seek support from PACE in choosing cost effective and proper hardware for their purpose. Pricing is current as of July, 2019.

Going into CODA, PACE is moving to the "Cascade Lake" CPU technology from Intel.  We're also moving to a 100-gigabit high-performance network, HDR100 Infiniband, and 10-gigabit ethernet on all compute nodes.  We're also evaluating a move from SSD to next-generation NVMe technology for system drives.

Participants are strongly encouraged to select from the following configuration choices.  Reducing the number of configurations helps PACE staff provide efficient service, and focus efforts on projects of strategic value.

192 GB Compute node - $7,200

  • dual-socket, 12-core Intel Xeon Gold 6226 "Cascade Lake" @ 2.7Ghz (24 cores total)
  • 192 GB DDR4-2933 Mhz memory
  • HDR100 Infiniband card
  • port on the PACE HDR Infiniband switch
  • 10-gigabit Ethernet cabling
  • shipping, installation and burin-in testing
  • 5-year next-business-day on-site warranty

384 GB Compute node - $8,600

  • same configuration as the 192 GB Intel node, just more memory

768 GB Compute node - $10,000

  • same configuration as the 192 GB Intel node, just more memory

 

192 GB Compute node w/ local disk - $9,200

  • same configuration as the 192 GB Intel node, plus (qty 4) 2TB SAS drives

384 GB Compute node w/ local disk - $10,200

  • same configuration as the 384 GB Intel node, plus (qty 4) 2TB SAS drives

768 GB Compute node w/ local disk - $11,900

  • same configuration as the 768 GB Intel node, plus (qty 4) 2TB SAS drives

 

NOTE: Due to licensing restrictions from nVidia, PACE cannot support consumer-grade nVidia GPUs (e.g. the GTX or Titan line).  For those desiring single-precision GPU performance (e.g. AI or machine learning workloads), we do support the nVidia T4 GPU.

192 GB Compute node w/ single precision GPU - $16,000

  • same configuration as the 192 GB compute node, plus (qty 4) nVidia T4 GPUs

384 GB Compute node w/ single precision GPU - $16,900

  • same configuration as the 384 GB compute node, plus (qty 4) nVidia T4 GPUs

768 GB Compute node w/ single precision GPU - $18,600

  • same configuration as the 768 GB compute node, plus (qty 4) nVidia T4 GPUs

 

192 GB Compute node w/ double precision GPU - $19,700

  • same configuration as the 192 GB compute node, plus (qty 2) nVidia v100 GPUs​

384 GB Compute node w/ double precision GPU - $20,600

  • same configuration as the 384 GB compute node, plus (qty 2) nVidia v100 GPUs​

768 GB Compute node w/ double precision GPU - $22,500

  • same configuration as the 768 GB compute node, plus (qty 2) nVidia v100 GPUs​

 

Storage - $80 / TB / year

  • provisioned from shared GPFS filesystem
  • may be easily expanded on demand
  • smallest increment is 1TB
  • multiple years may be paid up front (i.e. $240 / TB for 3 years)
  • includes nightly backups

[ Return to top ] 


"What is the schedule for procurements?"

PACE equipment and staff will be moving to the new CODA facility in 2019.  While we will do our best to limit the impact of this transition on the GT research community, it will require adherence to a proscribed schedule of procurements.  Note that storage purchases may occur outside of this schedule.

These schedules are based on the best available information at this time, and reflect a realistic timeline that is based on experiences of previous orders.

 

FY20-Phase1 - to be deployed directly to CODA.  This is a tentative schedule and will be refined to adjust to the completion of the ERP transition, and the opening of FY20 procurements.
deadline description notes
6/10/2019 intent period Please contact PACE via pace-support@oit.gatech.edu to declare your intent to participate in this round.
6/17/2019 actionable requests Information such as preliminary configuration and/or budget, account number(s), spend/finance approvers, cluster name, and if the compute nodes should be shared.
6/19/2019 preliminary quote PACE will obtain a preliminary quote based on actionable requests received by the deadline.
6/26/2019 configuration adjustments Any configuration adjustments needed based on current prices.
6/28/2019 final quote PACE will obtain a final quote
7/5/2019 faculty approvals Upon receipt of final quote, PACE will send a form letter to all participants with "to the penny" pricing and a detailed description.  Confirmation from each participant is required to proceed with the procurement.
7/8/2019 enter requisition PACE will enter the requisition into BuzzMart for financial approvals
7/29/2019 release PO estimated point at which the formal purchase order will be transmitted to the vendor
9/23/2019 manufacturing estimated time to manufacture and assemble systems by the vendor
9/30/2019 shipping  
10/7/2019 installation estimated completion of physical installation by the vendor
11/4/2019 acceptance testing PACE will perform acceptance testing to ensure proper operation.  Upon completion, resources will be made "ready for research"
 

[ Return to top ] 


"What do I get in return?"

The advantage of the federated model is that everyone benefits from the Institute's commitment to pre-load infrastructure and support.  This applies to every participant who chooses any of the first 4 options described above.  These benefits include:

 
  • lower direct costs to participants for HPC equipment purchases by leveraging shared resources
  • guidance in developing system specifications consistent with the federated architecture
  • full life-cycle procurement management
  • hosting in a professionally managed data center with a 24/7 operations staff
  • racks, power and cooling to meet high-density demands
  • installation management
  • acceptance testing in accordance with both PACE and end-user requirements
  • secure high-speed scratch storage
  • head node for login access
  • a small home directory (bulk storage is funded by the faculty member)
  • commodity Ethernet networking (Infiniband, if desired, is funded by the faculty member)
  • back-up and restore
  • queue management
  • system administration
  • software and compiler administration (loads, updates, licensing)
  • hardware fixes
  • the dedicated support team manages all aspects of the cluster
  • shared access to recent architecture

 

[ Return to top ]