Frequently Asked Questions

General (8)

What is Pace?

PACE (a Partnership for an Advanced Computing Environment) is a collaboration between Georgia Tech faculty and the Office of Information Technology (OIT) focused on HPC.  This partnership is made possible as part of a long-term commitment by Georgia Tech to create an exemplary environment  to advance Georgia Tech's leadership position in innovative research.

PACE provides a core suite of HPC services and resources available at no direct cost to faculty participants.  These resources include:

  • Infrastructure - data center space, power, cooling, racks, and all aspects of 24x7 data center operations
  • Commodity Services - ethernet, high-speed scratch storage, back-ups, recovery, security
  • Software - compliers, applications, license management
  • Technical Services & Support - user support, hardware management, consulting, procurement
  • FoRCE - scalable cluster of shared nodes managed by faculty

PACE also provides faculty the opportunity to leverage their investments in exclusively dedicated nodes by employing a federated model  where dedicated nodes purchased by faculty are integrated into the PACE set of core services.  Thus, participating faculty avoid the costs associated with these core services.

Find out how to get started at PACE at our participation page.

What is Inchworm?

The PACE federation of clusters includes all HPC clusters that are integrated into PACE's suite of core services and resources.  These include:

  • FoRCE, a shared pool of community nodes immediately available for access
  • Several clusters of nodes that share job assignments with the FoRCE.
  • Several clusters of nodes dedicated to the exclusive use of the faculty who purchase the nodes.

PACE's suite of core services and resources provided to the federation of clusters include:

  • Baseline pool of FoRCE nodes
  • Infrastructure - data center space, power, cooling, racks, and all aspects of 24x7 data center operations
  • Commodity Services - ethernet, high-speed scratch storage, back-ups, recovery, security
  • Software - compliers, applications, license management
  • Technical Services & Support - user support, hardware management, consulting, procurement

See Participants and Resources for more information.

What are the different ways to participate and the associated services?

Services range from a shared community cluster, known as FoRCE, to clusters dedicated to individual researchers.  Common infrastructure and system administration are provided by PACE.  Faculty contribution depends on which participation option best fits individual faculty needs.  Please see Participation for more details.

How and when is Inchworm upgraded?

Major maintenance of the PACE federation of clusters occurs quarterly.  The maintenance schedule is approved by the Faculty Governance Committee and subject to change.

Generally, significant hardware acquisitions occur annually after the Supercomputing Conference (SC) in November.  Waiting until after the SC conference provides insights into emerging technologies and peer trends which helps guide technical specifications for the upcoming procurement.

If a faculty member has funding that needs to be spent on a different timetable, we will work with them to help them take advantage of any currently negotiated contracts and discounts.

Please see Policy for detailed information regarding maintenance procurement schedules.

How much does it cost and who pays for what?

Costs vary based upon mainstream and experimental technology trends, procurement cycle, size of order, and economic conditions.  Researcher contribution, either through start-up package funding or sponsored awards, depends on the chosen level of participation.  In general, participants will cover the cost of compute nodes, bulk "project directory" storage and networking beyond commodity gigabit ethernet.  Researchers who opt to purchase nodes for shared use in the FoRCE Cluster will have priority use of those nodes plus shared access to the entire FoRCE Cluster.  Researchers who opt to pay nothing can be granted shared access to the FoRCE Cluster for a fixed number of CPU hours, as allocated by the Faculty Governance Committee. Other expenses are considered infrastructure-related, and are covered by the Institute.

Please see Participation for more details.

What are the CSR's role in PACE?

PACE is a consolidated family of resources whose sole purpose is the support of computationally intensive research on behalf of Georgia Tech.  Departmental IT Directors and Computer Support Representatives (CSR's) are responsible for supporting a wide spectrum of IT services for their respective areas.  CSR's are encouraged to add PACE perspective while advising their Faculty on a solution that best meets individual needs. PACE team is always ready to support CSR's with their questions related to PACE resources and HPC in general.

Where do I email support questions again?

Please see our Support page for contact information.

I've got pending research that requires new HPC resources quickly. What do I do?

The need for stand-alone deadline driven purchases compatible with the Inchworm infrastructure should be communicated to the PACE team as quickly as possible.

Why do my jobs that use mpiexec take fail to complete until the walltime expires?

mvapich's mpiexec has some hard-coded dependiences that reference an expired version of torque. Please use mpirun, or use openmpi.
 

Why don't I see my job outfile from job X?

First, please check if your job was rejected (e.g. wallclock higher than limit) using 'checkjob'; .o/.e files are not created for any problematic jobs. 
 
If that is not the case, and you are running many jobs which are creating many outfiles at the same location over a very short period of time, then please use a faster file-system (like scratch) to receive your outfiles. You can also direct the output from the program-part of your pbs script to write to a particular source (ie: instead of running: 'mpiexec this', run 'mpiexec this > /your/project/directory/this.out').
 

Why do I sometimes get an error when I try to delete a job using qdel?

qdel commands can occasionally fail, but will probably work when you try again. You can also use the 'canceljob' command.

 

Why is it, when I try to start a job, I get a "limit reached"-type error?

This could mean you have over 3000 jobs listed under qstat (and you've been submitting jobs using qsub), or it means the 15000 job limit has been reached (and you've been submitting jobs using musb). Please use the qstat (qsub) or showq (msub) command to determine the number of free slots available.
 
 

Why am I'm seeing a "RM Failure" error when my deferred job tries to start?

Requests by the scheduler to the resource manager will occasionally fail, in cases like this the system will retry every ten minutes until successful.
 

Why am I limited to 2000 idle/queued jobs (additional jobs are deferred)?

 This is to prevent someone from accidentally blocking other users jobs from starting. 

Why am I limited to 3000 running jobs?

This is to prevent someone from accidentally making the system unresponsive.

Why am I limited to a size of 2000 when submitting a job-array?

This is to prevent someone from accidentally blocking all job submissions made by other users.
 

How do I use custom variables to define the name of my job?

Define your -v varA=valA,varB=valB, etc list as before, but define -N within your pbs script

Why am I getting 'command not found' and module load errors when my job starts?

This was due to a problem in a profile.d file and should now be corrected. You can add  'source /etc/profile.d/modules.sh' to your pbs script, before the module load commands  to work around the issue, but please report this problem to PACE.

Why do my jobs die, without warning, shortly after I submit them using qsub?

This usually means that the system has determined that your jobs can never start. Try resubmitting the same jobs using msub to find out why, or use checkjob.
 

Why can’t I submit jobs from compute nodes?

Enough simultaneous qsub requests will severely degrade system performance.  Please use msub if you need to submit jobs from compute nodes.
 

My job arrays all have different PBS_JOBIDs; How can I group the output files?

Use the variable '$MOAB_JOBARRAYINDEX' as part of your out-file name, and do not use $PBS_JOBID. You will need to use different out-file names for different job arrays if you are outputting them all to the same folder.

Why can't I submit job-arrays using qsub anymore?

The new (and previous) versions can become unresponsive if large job-arrays are submitted via qsub. Please use 'msub' to submit job-arrays. Also substitute $MOAB_JOBARRAYINDEX with $PBS_ARRAYID inside your PBS scripts. The use of '-t' parameter in msub is the same as qsub, except for the name of this variable.

Why do my jobs that use mpiexec take fail to complete until the walltime expires?

mvapich's mpiexec has some hard-coded dependiences that reference an expired version of torque. Please use mpirun, or use openmpi.
 

Why don't I see my job outfile from job X?

First, please check if your job was rejected (e.g. wallclock higher than limit) using 'checkjob'; .o/.e files are not created for any problematic jobs. 
 
If that is not the case, and you are running many jobs which are creating many outfiles at the same location over a very short period of time, then please use a faster file-system (like scratch) to receive your outfiles. You can also direct the output from the program-part of your pbs script to write to a particular source (ie: instead of running: 'mpiexec this', run 'mpiexec this > /your/project/directory/this.out').
 

Why do I sometimes get an error when I try to delete a job using qdel?

qdel commands can occasionally fail, but will probably work when you try again. You can also use the 'canceljob' command.

 

Why is it, when I try to start a job, I get a "limit reached"-type error?

This could mean you have over 3000 jobs listed under qstat (and you've been submitting jobs using qsub), or it means the 15000 job limit has been reached (and you've been submitting jobs using musb). Please use the qstat (qsub) or showq (msub) command to determine the number of free slots available.
 
 

Why am I'm seeing a "RM Failure" error when my deferred job tries to start?

Requests by the scheduler to the resource manager will occasionally fail, in cases like this the system will retry every ten minutes until successful.
 

Why am I limited to 2000 idle/queued jobs (additional jobs are deferred)?

 This is to prevent someone from accidentally blocking other users jobs from starting. 

Why am I limited to 3000 running jobs?

This is to prevent someone from accidentally making the system unresponsive.

Why am I limited to a size of 2000 when submitting a job-array?

This is to prevent someone from accidentally blocking all job submissions made by other users.
 

How do I use custom variables to define the name of my job?

Define your -v varA=valA,varB=valB, etc list as before, but define -N within your pbs script

Why am I getting 'command not found' and module load errors when my job starts?

This was due to a problem in a profile.d file and should now be corrected. You can add  'source /etc/profile.d/modules.sh' to your pbs script, before the module load commands  to work around the issue, but please report this problem to PACE.

Why do my jobs die, without warning, shortly after I submit them using qsub?

This usually means that the system has determined that your jobs can never start. Try resubmitting the same jobs using msub to find out why, or use checkjob.
 

Why can’t I submit jobs from compute nodes?

Enough simultaneous qsub requests will severely degrade system performance.  Please use msub if you need to submit jobs from compute nodes.
 

My job arrays all have different PBS_JOBIDs; How can I group the output files?

Use the variable '$MOAB_JOBARRAYINDEX' as part of your out-file name, and do not use $PBS_JOBID. You will need to use different out-file names for different job arrays if you are outputting them all to the same folder.

Why can't I submit job-arrays using qsub anymore?

The new (and previous) versions can become unresponsive if large job-arrays are submitted via qsub. Please use 'msub' to submit job-arrays. Also substitute $MOAB_JOBARRAYINDEX with $PBS_ARRAYID inside your PBS scripts. The use of '-t' parameter in msub is the same as qsub, except for the name of this variable.

Technical (11)

How can I learn more about utilization of clusters by users/groups ?

PACE offers the following tools for inspecting performance/utilization/status of PACE managed clusters. Please refer to user guide for more details on Ganglia and Metrics:

  • Ganglia
    Cluster status, CPU and memory utilization, offline nodes
  • Metrics
     Detailed, sortable useage statistics per user/group/queue
     
  • MRTG
    Network Performance

 

What are those "data" and "scratch" folders in my home directory?

In your home directory, there are two symbolic links - data and scratch.  The data symbolic link points to your project directory space for long term storage of data sets.  The scratch symbolic link points to your space on the high-performance scratch storage.  As part of your job submission file, you can make additional directories within your scratch space and copy input files from your project directory into the newly created sub-directory on the scratch.  During the execution of your job, operate on the copy within the scratch space.  When your calculations are complete, copy needed files back to your project directory space and remove the remaining files from the scratch space.

Remember, the scratch space is limited and not intended to hold data for the long term.  We implement automated removal of "old" files (> 60 days old) from the scratch space each week. In addition, we will apply 10TB hard quotas per user effective after 15/10/2012. We do not perform backups on the scratch storage, but do for the project directory and home directory storage.

Some clusters (Joe, Atlantis and parts of Atlas) lack Infiniband connectivity.  Therefore access to the scratch storage utilizes the commodity ethernet network.  Because of this, you may actually see better performance by using /tmp on machines with local disks.  HOWEVER, most of our newer systems lack local disks, so the scratch is probably your best bet.

  • Atlas cluster - use the scratch on the infiniband connected nodes, /tmp otherwise.
  • Atlantis cluster - all nodes are diskless and lack infiniband.  use the scratch.
  • Joe cluster - all nodes lack infiniband.  The old Sun machines have local disks, so use /tmp.  The new nodes from Penguin do not have disks, so use the scratch.

Use the following bit of shell code in your job submission script to determine if you have a local disk:

if [ `mount | grep /tmp` ]; then
  echo "use /tmp"
else
  echo "use the high-performance scratch storage"
fi

What about the CPUs on this node?

This command may help:

foo@joe98 ~> cat /proc/cpuinfo
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 65
model name	: Dual-Core AMD Opteron(tm) Processor 2222
stepping	: 3
cpu MHz		: 3015.524
cache size	: 1024 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy
bogomips	: 6038.61
TLB size	: 1088 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 65
model name	: Dual-Core AMD Opteron(tm) Processor 2222
stepping	: 3
cpu MHz		: 3015.524
cache size	: 1024 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy
bogomips	: 6030.07
TLB size	: 1088 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

processor	: 2
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 65
model name	: Dual-Core AMD Opteron(tm) Processor 2222
stepping	: 3
cpu MHz		: 3015.524
cache size	: 1024 KB
physical id	: 1
siblings	: 2
core id		: 0
cpu cores	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy
bogomips	: 6030.07
TLB size	: 1088 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

processor	: 3
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 65
model name	: Dual-Core AMD Opteron(tm) Processor 2222
stepping	: 3
cpu MHz		: 3015.524
cache size	: 1024 KB
physical id	: 1
siblings	: 2
core id		: 1
cpu cores	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy
bogomips	: 6030.07
TLB size	: 1088 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

foo@joe98 ~>

 

 

 

How much RAM is on this node?

This command may help:

foo@joe98 ~> cat /proc/meminfo
MemTotal:     16419200 kB
MemFree:      15035800 kB
Buffers:        239928 kB
Cached:         283524 kB
SwapCached:       2120 kB
Active:         706044 kB
Inactive:       114520 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     16419200 kB
LowFree:      15035800 kB
SwapTotal:     2096440 kB
SwapFree:      1899200 kB
Dirty:             316 kB
Writeback:           0 kB
Mapped:         303940 kB
Slab:           540140 kB
CommitLimit:  10306040 kB
Committed_AS:   504724 kB
PageTables:       2516 kB
VmallocTotal: 536870911 kB
VmallocUsed:      1320 kB
VmallocChunk: 536869559 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB
foo@joe98 ~>

 

 

 

What's eating up all my disk space?

This rather long command might be helpful:

foo@joe ~> cd; du -ks * | sort -nr | cut -f2 | xargs -d 'n' du -sh

Or, you could simply list files that are Gigabytes in size: 

foo@joe ~> cd; du -sh * | grep G

Note that this also lists all the filenames that include the letter 'G', regardless of their size.

How do I see what processes I have running on the joe cluster?

This ain't pretty, but works if you want to see your running processes on a cluster, e.g. the FoRCE:

/usr/local/sbin/dsh -F 20 -g force -- "ps aux | grep `whoami`" | grep -v 'ps aux' | grep -v "grep `whoami`"

Please replace 'force' (passed by -g) with a corresponding group name, as listed in /etc/dsh/group, for other clusters.

 

Is there a way to allow my interactive sessions to persist as I travel between my home and lab computers?

Yes, please refer to the following links for more information.

GNU Screen: an introduction and beginner's tutorial:

http://www.kuro5hin.org/story/2004/3/9/16838/14935

 

from the Linux  UsersGroup at Georgia Tech:

http://www.lugatgt.org/articles/gnu_screen/

http://www.lugatgt.org/articles/screen/

 

from Wikipedia:

http://en.wikipedia.org/wiki/GNU_Screen

 

 

How do I get system email sent elsewhere?

If you are intereted in PBS-issued emails only, you can specify your email address in the PBS submit script, followed by the "#PBS -M":

#PBS -M your_email_address

If you would like *all* system emails forwarded, then you create a .forward file in your home directory:

foo@pacemaker ~> cd
foo@pacemaker ~> echo your_email_address > .forward

 

What to do if you have trouble transferring files?

Problems: dropping speed, hung scp transfers

These intermittent problems have been reported before, but we have been unable to reproduce them consistently in order to determine the cause. In the past, though, usually on the 4th or 5th attempt, the file would transfer successfully.

As possible workarounds, please try the following:

1) try both pushing or pulling the file from both your desktop and from joe (that is, try the scp command from both your machine and from joe).

2) try the same above, but using your acme.gatech.edu account as an intermediary transfer location. For example,

on the server...

foo@joe wrk1> ls -l
total 7076
-rw-r--r--  1 foo rnoc 7232850 Dec 22 15:13 some_file
foo@joe wrk1> scp some_file acme.gatech.edu:
foo@acme.gatech.edu's password: 
foo@joe wrk1> ssh acme.gatech.edu
foo@acme.gatech.edu's password: 
{acmex:foo:127} ls -l some_file
-rw-r--r--   1 foo     gtperson 7232850 Dec 22 15:16 some_file
{acmex:foo:128} pwd
/prism/39/foo

 

on your desktop...

desktop:wrk2 foo$ ls
desktop:wrk2 foo$ scp foo@acme.gatech.edu:/prism/39/foo/some_file .
foo@acme.gatech.edu's password: 
some_file                        100% 7063KB  63.1KB/s   01:52    
desktop:wrk2 foo$ ls -l
total 14128
-rw-r--r--  1 foo  staff  7232850 Dec 22 15:22 some_file
 

What id I'm having permission/password problems?

PACE clusters use the standard "GT Account" provided to all GT faculty, staff and students.  For external collaborators, we can provision guest accounts created by their GT sponsor.

Guest accounts and password resets can be resolved by using the Georgia Tech Personal Account Self-Service Portal (aka passport).

If you still have problems with your password, please see your local Computer Support Representative (CSR), or visit the Technology Support Center.

How can I get general information about all clusters?

OIT offers two webpages, namely Ganglia and Metrics, to provide users with any kind of information they might need about the clusters. These webpages can only be browsed on campus, or via VPN.

Ganglia: The main page of Ganglia provides two graphs for CPU and Memory utilization for the past hour, for each cluster. You can get historical information up to a year from the menu titled "Last" (see figure below). To get more detailed (i.e. per-node) information, you can click to the cluster title, or any of the graphs. The workload on each node are color coded, e.g., nodes that use almost 100% of CPUs will appear red. If you submitted a job to a cluster and it is not allocted for a long time, you can always check the cluster utilization from this webpage and see how many nodes are busy/idle. If the cluster looks idle and your jobs is not still being allocated, then please check your PBS parameters for typos or whether 

Can my non-GT collaborator access my cluster?

 The short answer is, yes.

A guest account is required.  These may be created on passport.gatech.edu.  Generally, these credentials are created by the faculty member sponsoring the remote collaborator.  Once these credentials are created, PACE staff can enable access to PACE clusters as we do for other GT Accounts.  When creating these credentials, be  sure to enable "basic authentication."  There are limits on the duration that these accounts may be enabled, but they may be renewed by the sponsor on the Passport site.  More details on guest accounts can be found on the Passport site.

We can also use this mechanism to prolong the access of student collaborators as they graduate.

Please also see our FAQ entry on accessing PACE clusters from off campus.

What do you recommend for file transfer from a Windows machine to the clusters?

In general, HPC resources managed by PACE are only accessiable via the SSH protocol.  Any client software that implements this standard should be usable. If you are off-campus, make sure you have an active VPN connection first.

Datamover nodes: All pace users have access to a dedicated file transfer server equipped with high speed network interfaces. Use of this node not only will give better performance for file tranfers, but also reduce unnecessary i/o on headnodes, reducing impact on other pace users. To access, please log directly into iw-dm-4.pace.gatech.edu, or rich116-d39-35.pace.gatech.edu.

Windows: For clients running the Microsoft Windows operating system, Georgia Tech maintains a site license for SecureCRT and WinSCP. These packages are available from software.oit.gatech.edu.  Also know to work is Putty.  Please see your local CSR for IT support on the installation and operation of these packages.

MacOS: MacOS users can easily mount their home directory using Macfusion using the "SSHFS" protocol and easily drag&drop files. A combination of MacFuse and sshfs is also suggested in several online forums, but we do not recommend this solution at the moment due to a current bug in sshfs.

Linux: Linux users only need to type: sftp://@.pace.gatech.edu in the address bar of their favorite file browser. That is, of course, after replacing and with proper values!

Can I access PACE from off-campus when I am at home?

Yes, you have two options, the preferred method is connecting via the campus VPN.  For more information on using the VPN, please refer to http://oit.gatech.edu/service/vpn/vpn-service-remote-access.


A second option is to login to acme first.

macbook:~ user$ ssh acme.gatech.edu
Password:

[user@acmex ~]$ ssh force.pace.gatech.edu

user@force.pace.gatech.edu's password:
Last login: Tue Sep 15 18:22:18 2010 from acmex.gatech.edu

RHN kickstart on 2010-04-27

[user@force ~]$

 

 

How do I stop compute nodes from asking for my password?

Head nodes are accessible from on-campus networks (and in some cases only specific networks or IP Addresses) via SSH. If you are off-campus, make sure you have an active VPN connection first.

Compute nodes are only accessible from other compute nodes and head nodes via either SSH or RSH.  However, you will only be able to login to a given compute node if you have an active job running on that node.   If the scheduler detects that your jobs on a given compute node have completed, any "leftover" or "derelict" processes left running on that compute node are subject to termination.

RSH is a passwordless protocol, and should require no user action to work.  If you encounter trouble using rsh, please file a trouble ticket.

SSH encryption keys are normally created with your user account, and eliminate the need to supply a password when logging in via SSH. You can issue the command  "ls -al ~/.ssh" to see if you already have a preconfigured SSH. If that's not the case, you may either file a remedy ticket, or alternatively  follow the procedure below:

Starting out with no .ssh directory, you can do the following (just hitting Return when ssh-keygen prompts you for a response): 

[user@force ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/nv/hp16/user/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /nv/hp16/user/.ssh/id_rsa.
Your public key has been saved in /nv/hp16/user/.ssh/id_rsa.pub.
The key fingerprint is:
86:f1:d7:54:f4:05:8e:d7:68:26:0b:47:16:86:2e:db user@joe.pace.gatech.edu

[user@force ~]$ cd ~/.ssh

[user@force .ssh]$ ls
id_rsa  id_rsa.pub

[user@force .ssh]$ cat id_rsa.pub >> authorized_keys2
      [user@force .ssh]$ ls
      id_rsa  id_rsa.pub   authorized_keys2
 

 

 

How do I use cluster GUI apps (from off-campus)?

First, you need a X11 client and a terminal on your local machine. On windows, you could use Xwin32 (available for free in http://software.oit.gatech.edu), XLiveCD or Xming. On MacOSX, you could install XQuartz and use the Terminal application located in Application->Utilities. Linux already comes with X11, users can select from several options including “gnome-terminal” (for GNOME) and “konsole (for KDE)”.

Then, you ask SSH client to forward X-Window on your screen, using the -X option (or similarly -Y in recent versions):

$ ssh -X 

If you are off-campus, make sure you have an active VPN connection first. Another option is to use the ssh X-Window options above to login to "acme.gatech.edu", which is open to the world.

$ ssh -X username@acme.gatech.edu

first, and then again, from acme to your cluster's head node. 

Note: One user reported that -X works from local machine to acme, but not from acme to a cluster node. Using -Y instead seems to solve this problem. Yet again, the supported and *encouraged* way it to use VPN and ssh to the head node directly using -X.

 

 

I already know how to use MATLAB, should I still read this FAQ entry? (Hint: yes...)

 

Please read this warning if you are submitting Matlab jobs on any of the queues... Thanks!

 


 

 

 

 

How do I fix memory limit errors when running ABAQUS?

How can I ask for a particular node and/or property?

 PBS allows for asking for particular compute nodes that share a set desired properties.

  • If you need a specific set of nodes (e.g. iw-h31-9, iw-h31-10, iw-h31-11), then you can add them (using the '+' sign) directly in the 'nodes' parameter:

 #PBS -l nodes=iw-h31-9+iw-h31-10+iw-h31-11

  • If you need any node that has a specific property (nvidiagpu, number of cores, SSE capabilities, etc), then you can use tags, e.g.:

 #PBS -l nodes=2:ppn=4:nvidiagpu

 

Common tags include:

core8 (8 core node)

 

core24 (24 core node)

 

core48 (48 core node)

 

core64 (64 core node)

 

intel

 

amd

 

nvidiagpu

 

amdgpu

 

localdisk (local disk)

 

ib   (infiniband network)

 

ssd (solid state local disk)

 

sse3

 

sse41

 

fma4 (FMA4 capability)

 

 

PBS cannot allocate resources, no matter what I ask for!

We have seen this issue happening when the PBS script includes memory request on the same line as the processor request. For example:

#PBS -l nodes=12:ppn=4:mem=8GB

Please try placing the memory to a separate line:

#PBS -l nodes=12:ppn=4
#PBS -l mem=8GB

 

PS: This issue reveals itself with the following error message:

qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes

 

Can I submit multiple PBS jobs? If so, how?

Yes! You can prepare a "template" PBS script and submit it multiple times, each with different argumetns, using a very simple script. See this page for instructions (find the subsection "Submitting Multiple Jobs Quickly"). This is a much easier and faster method than preparing a PBS script and submitting it manually for each and every case!

Do you have any nodes with GPUs? How can I use them?

The force queue now includes a number of nodes with GPUs. All you need to do is prepare a PBS submit script that includes the line:

#PBS -l nodes=2:ppn=8:gpu

to select 2 nodes, each with 8 cores and a GPU.
 

PS: If this does not work for you, try placing 'gpu' in a separate line:

#PBS -l nodes=2:ppn=8
#PBS -l gpu

 

 

How do I check the status of a job?

 Once we have a job id (which can be obtained using qstat), we can find out a little more about that job like this:

[root@force ~]# checkjob 137500
job 137500

AName: CJV.data.970.20
State: Running 
Creds:  user:myusername  group:mygroupname  class:force
WallTime:   22:56:44 of 8:08:00:00
SubmitTime: Mon Nov 22 10:37:03
  (Time Queued  Total: 1:24:04  Eligible: 1:22:20)

StartTime: Mon Nov 22 12:01:07
Total Requested Tasks: 1

Req[0]  TaskCount: 1  Partition: repace  
Memory >= 5200M  Disk >= 0  Swap >= 0
Opsys: ---  Arch: ---  Features: ---
Dedicated Resources Per Task: PROCS: 1  MEM: 5200M
NodeCount:  1

Allocated Nodes:
[iw-h41-17a.pace.gate:1]

StartCount:     1
Partition List: repace
StartPriority:  1832
Reservation '137500' (-22:56:57 -> 7:09:03:03  Duration: 8:08:00:00)

Some interesting items in this output are:

  • AName - the name of the job supplied by the user in their job submission file
  • State: valid states are Running, Idle, UserHold, SystemHold, Deferred, BatchHold and NotQueued
    • Running - the job has been issued to a compute node and is executing
    • Idle - the job is awaiting resources to become available before it can execute
    • UserHold - the job is not being considered for execution at the request of the user
    • SystemHold - like UserHold, except the hold has been placed by an administrator
    • Deferred - a temporary hold, used by the scheduler when it has been unable to start the job after a number of attempts.  Usually, this results from some policy violation such as exceeding the maximum allowed wall clock time. See this FAQ entry for more information. 
    • BatchHold - the scheduler has failed in multiple attempts to start the job.  Usually, this is the permanent state Deferred jobs transition into.
    • NotQueued - something went seriously wrong with this job, and it has only partially been entered into the scheduler.  Contact support for assistance.
  • Creds: credentials of the job
    • user - the username of the account that submitted the job, in this case "myusername"
    • group - the primary unix group of the account that submitted the job, in this case "mygroupname"
    • class - the queue into which the job has been submitted
  • WallTime: elapsed and maximum time this job is allowed to run.  In this case, the job has been executing for 22 hours, 56 minutes, 44 seconds.  If it doesn't complete before a total of 8 days, 8 hours, it will be terminated.
  • SubmitTime: when the job was submitted to the queue
  • Time Queued: how long the job waited in the queue for resources to become available
  • StartTime: when the job began execution
  • Total Requested Tasks: the total number of CPUs the job has requested
  • Allocated Nodes: a list of compute nodes this job is utilizing.  In this case, one processor on node iw-h41-17a.

 

How do I tell what's running on this node?

This is a two step process.  The first part is a command to show all of the jobs running on a given node, and the other is a command to show the details of a given job.  For the second part, see our FAQ entry on checking the status of a job.

For instance, to find out about the node iw-h41-17a, use the node_status command (output formatted for better web display):

[root@force ~]# /usr/local/sbin/node_status iw-h41-17a
iw-h41-17a.pace.gatech.edu
     state = job-exclusive
     np = 24
     ntype = cluster
     jobs = 0/137499.repace.pace.gatech.edu, 1/137500.repace.pace.gatech.edu,
            2/138400.repace.pace.gatech.edu, 3/138400.repace.pace.gatech.edu,
            4/138400.repace.pace.gatech.edu, 5/138400.repace.pace.gatech.edu,
            6/138400.repace.pace.gatech.edu, 7/138400.repace.pace.gatech.edu,
            8/138400.repace.pace.gatech.edu, 9/138400.repace.pace.gatech.edu,
            10/139359-69.repace.pace.gatech.edu, 11/139359-70.repace.pace.gatech.edu,
            12/139359-71.repace.pace.gatech.edu, 13/139359-72.repace.pace.gatech.edu,
            14/139359-73.repace.pace.gatech.edu, 15/139359-74.repace.pace.gatech.edu,
            16/139359-75.repace.pace.gatech.edu, 17/139359-76.repace.pace.gatech.edu,
            18/139359-77.repace.pace.gatech.edu, 19/139359-78.repace.pace.gatech.edu,
            20/139359-79.repace.pace.gatech.edu, 21/139359-80.repace.pace.gatech.edu,
            22/139359-81.repace.pace.gatech.edu, 23/139359-82.repace.pace.gatech.edu
     status = opsys=linux,uname=Linux iw-h41-17a.pace.gatech.edu 2.6.18-194.el5 #1 
            SMP Tue Mar 16 21:52:39 EDT 2010 x86_64,sessions=943 997 6286 7694
            16223 16301 16554 16804 17044 17270 17526 17766 18019 18246 18496 18737
            18982 19343 25479 29192,nsessions=20,nusers=6,idletime=1178489,
            totmem=66012996kb,availmem=55511080kb,physmem=66012996kb,ncpus=24,
            loadave=16.02,netload=40780188503,state=free,
            jobs=137499.repace.pace.gatech.edu 137500.repace.pace.gatech.edu
            138400.repace.pace.gatech.edu 139359-69.repace.pace.gatech.edu
            139359-70.repace.pace.gatech.edu 139359-71.repace.pace.gatech.edu
            139359-72.repace.pace.gatech.edu 139359-73.repace.pace.gatech.edu
            139359-74.repace.pace.gatech.edu 139359-75.repace.pace.gatech.edu
            139359-76.repace.pace.gatech.edu 139359-77.repace.pace.gatech.edu
            139359-78.repace.pace.gatech.edu 139359-79.repace.pace.gatech.edu
            139359-80.repace.pace.gatech.edu 139359-81.repace.pace.gatech.edu
            139359-82.repace.pace.gatech.edu,varattr=,rectime=1290526391

Some explanation of the output may be in order.

  • state - valid states are down, offline, free and job-exclusive
    • job-exclusive - the node is completely filled with jobs.
    • free - slightly misleading.  Some jobs may be executing, but not all processors are in use.
    • offline - administratively down, usually means that PACE staff are making repairs.
    • down - the scheduling services are not running on this node.  usually means the node is powered off.
  • np - number of processors.  In the example above, iw-h41-17a has 24 processors.
  • ntype - it will always be "cluster" in our environment.  Move on, nothing to see here.
  • jobs - a  list of job id's, and which logical processor to which they are assigned. 
  • status - lots of status information, including job id's, all on a single line to facilitate easy parsing with unix commands such as sed and awk.

 

What queues are available for job submission?

To find some basic queue information about a system, use the following command:

$qstat -Q

This command will be show a current utilization summary of every queue defined.

For more information, see the (rather outdated) User Guide page on Queue Names.

 

 

 

 

 

 

 

How do I delete jobs after they have been submitted to moab?

 

First, identify the JobID of the job you want to cancel.  In the example below, the '-w' argument to showq is used to only list jobs in the force queue.

[root@force ~]# showq -w class=force
active jobs------------------------
JOBID              USERNAME      STATE PROCS   REMAINING            STARTTIME
137077              user       Running    16  1:02:35:30  Sun Nov 21 23:32:35
1 active job            16 of 5216 processors in use by local jobs (0.31%)
                        137 of 255 nodes active      (53.73%)
eligible jobs----------------------
JOBID              USERNAME      STATE PROCS     WCLIMIT            QUEUETIME
0 eligible jobs   
blocked jobs-----------------------
JOBID              USERNAME      STATE PROCS     WCLIMIT            QUEUETIME
0 blocked jobs   
Total job:  1

After identifying which jobs you want to cancel, use the 'canceljob' command:

 

 

 

/usr/local/packages/moab/bin/canceljob 137007
 

 

 

 

Where can I check on the status of the campus network and systems?

OIT publishes a daily "Morning Report" of system and network activity on the main OIT web site, www.oit.gatech.edu.

We are have a "Message of the Day" web site in a pilot phase at motd.oit.gatech.edu.

What is "wallclock limit" and why are some jobs killed because of it?

Wallclock is the "maximum amount of real time during which the job can be in the running state." See here for more information.

Your wall clock limit is set in your pbs script with the walltime option:
 
#PBS -l walltime=HH:MM:SS
 

If your walltime request is too small, your job will be killed before it successfully completes; if it is too big, the scheduling algorithm may
make it wait in the 'Deferred' state for a longer period of time than necessary (or eventually put it in the 'BatchHold' state).

So, in order to stop your jobs from being killed, do your best to accurately estimate how long your job will take to run, and then use that value in your walltime request.

 

What does it mean when my job is 'Deferred'?

 

Please see this link for a list of job hold types and their explanations.

 

Can you increase my disk quota?

Disk quota on home directories is fixed at 5GB per user.  Space in project directories (~/data) is funded by faculty members who also set quota policy for thier users.  Project directory space on the FoRCE cluster is subject to space approved in your access proposal.  We will do our best to accomidate requests for additional space for FoRCE users as resources allow.

 

Why do my jobs have to wait so long in the queue?

Here are some possible scenarios:

No idle resources that match your requirements:

The cluster you are submitting to could be full. You might want to check the status of clusters and decide which queue to submit if you have access to more than one queue.

Unrealistically long walltime:

It is also a good practice to set the walltime only as long as needed, not more (if you have means to predict that). The PBS scheduler shows its best effort to keep the utilization rate high, and one strategy is allocating large jobs first, then filling-in the remaining spots with smaller jobs. The user manual explains this with a golf ball-sand analogy. Imagine you have a glass and you would like to fill it with a golf ball and some sand. If you put sand first, then you may not fit the golf ball later. But if you put the golf ball first, then you can easily pour sand into the empty space. If your job is actually a 'sand', but you misrepresent it as a 'golf ball' to the scheduler by providing an unrealistically large walltime, then you will need to wait in the queue until the previous golf ball in the glass to complete its job.

Some strategy in setting requirements might help:

> I've been noticing recently that when I submit a job on pace (~20 
> processors) it gets held up in the queue when it appears there are 
> many open processors. For instance, when I use the showq command it 
> looks like only about 10% of processors are being used (but perhaps 
> there is other activity I can't see).

The cluster statistics you get from the 'showq' command may include some nodes that you do not have access to.

For example, here is a current listing of nodes (and processors) for the 'joe' queue:

foo@pace1 ~>  /usr/local/sbin/dsh -F 20 -g joe 'cat /proc/cpuinfo | grep processor | wc -l'
iw-k35-32.pace.gatech.edu: 24
iw-k35-31.pace.gatech.edu: 24
iw-k35-19.pace.gatech.edu: 24
iw-k35-21.pace.gatech.edu: 24
iw-k35-34.pace.gatech.edu: 24
iw-k35-17.pace.gatech.edu: 24
...
...
 
The next time your jobs get held up in the queue, you could try cancelling them
/usr/local/packages/moab/bin/canceljob 

and resubmitting them with slightly altered resource requirements in your PBS submit script; for example, instead of

     #PBS -l nodes=4:ppn=48

(asking for 4 nodes with at least 48 free processors each, which is not available in the joe queue as shown above), you might try

     #PBS -l nodes=8:ppn=24

(asking for 8 nodes with 24 processors each). Depending on the cluster's current usage, a change like the one above could result in far less waiting time.

If no one else is currently using the nodes in the 'joe' queue, you are encouraged to use all the nodes and processors available, but when others are also making resource requests, generally, the more you ask for, the longer you have to wait.
 

How do I check my quota of filesytem space?

The 'quota' command is used for this.

[user@force ~]$ quota -v

Disk quotas for user user (uid 12345):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
iw-ss1.pace.gatech.edu:/hp16
                 294915  5242880 5242880      0       0       0       0       0

iw-ss2.pace.gatech.edu:/pf1
                      3  10485760 10485760    0       0       0       0       0
Some explanation is in order.  Units are in blocks of 1024 bytes.  In the above example, the user has almost 300MB of 5GB in use on hp16 and just a few bytes out of 10GB in use on pf1.

Software (7)

Compilers (2)

What are my choices of C++ compilers on joe again?

PACE provides licenses for many compiler options, either from campus site licenses or via recurring funding from the Institute.  We have various versions of GNU, Intel, Portland Group and NAG compilers installed, as well as builds of mpich, mpich2, and mvapich2.  Please see the software list for full details.

"How do I run C++ codes on the joe cluster?"

Here's a trivial example that runs through the basic process of compiling and submitting a serial job to the force cluster.  More information on using the Moab job scheduler can be found in our user guide: 

Job Submission

The Job Scheduler

 

[user@force ~]$ ls
hello.cpp  hello.pbs
 
[user@force ~]$ cat hello.cpp
#include  using namespace std;
int main(void) {
  cout << "hello, world";
  return 0;
}
 
[user@force ~]$ cat hello.pbs
#PBS -N Hello
#PBS -l nodes=1:ppn=1
#PBS -l walltime=5:00
#PBS -q force
#PBS -k oe
 
$HOME/hello
 
[user@force ~]$ g++ -o hello hello.cpp 
 
[user@force ~]$ ls
hello hello.cpp hello.pbs
 
[user@force ~]$ ./hello
hello, world
 
[user@force ~]$ msub hello.pbs
45796.repace.pace.gatech.edu
 
[user@force ~]$ cat ../Hello.o45796
hello, world

 

Can I take advantage of MATLAB's distributed computing toolbox when submitting jobs to the cluster?

Due to the cost and low use, we no longer maintain a Distributed Computing Server license for PACE. We will be happy to install DCS for any research groups who have purchased a license, however.

That said, the Parallel Computing Toolbox provides a similar functionality with the limitation to the number of cores and memory available on a single physical node.

What software do we have campus licenses for?

Georgia Tech maintains site license for many software packages.  Many of which are already installed on PACE managed clusters. Other HPC related software can be installed on a per-request basis.

All software and systems must be used in compliance with Georgia Tech's Computer and Network Usage and Data Access Policies. Please see our FAQ entries regarding cluster software restrictions for further details on software, and currently installed software

The source of GT licensed software is:

Also, here's a relatively short list of software available on PACE clusters.

How can I make my codes more efficient?

Thousands of people have earned doctoral degrees attempting to answer this question, we're certainly not going to be able to answer it in a FAQ!

However, here are some common things to start with: 

Compile your code with optimizations: Using "-O3" compiler optimization would provide significant speedup. 

Use numerical libraries: PACE clusters offer a number of numerical libraries for better performance and consistency. These libraries include fast and community-tested implementations for a large variety of computations, such as dense/sparse linear algebra, fourier transform, data processing, multimedia, and so on.

Use the scratch storage: If your code stores or manipulates large files, you may see benefit by utilizing the high-performance scratch storage. Here's how.  

Use GPUs: PACE clusters now include a bunch of nodes with GPUs... Use them! Here's how.

 

 

Where can I find the "Fastest Fourier Transform in the West"?

PACE provides licenses for many math library options, either from campus site licenses or via recurring funding from the Institute.  We have various versions of ACML, ATLAS, FFTW, NAG, GSL, PETsc, and the Intel MKL and IPP.  Please see the software list for full details

Where again is the official documentation for Moab and TORQUE?

See the "External Resources" section on our Resources page.

Are there any software restrictions for the clusters?

Software must be compatible with the supported Operating System and properly licensed.  HPC-relevant applications on OIT's Software Distribution website are readily available.  All software on PACE-adminstered clusters must be installed and used in complicance with Georgia Tech's Computer and Network Usage and Data Access Policies.

Please contact pace-support@oit.gatech.edu if you have any questions regarding specific applications, software licensing, and/or testing.

What software is available on the PACE clusters?

Software available to all Inchworm users includes:

  • abaqus-6.10
  • abaqus-6.7
  • abaqus-6.6
  • acml-4.4.0
  • acml-4.2.0
  • acml-4.1.0
  • ansys-12.1
  • ansys-11.0
  • atlas-3.9.23
  • atlas-3.8.2
  • atlas-3.6.0
  • cmake-2.6.4
  • cosmo140a
  • cosmo135a
  • dsh-0.25.7
  • dx-4.4.4
  • fftw-3.1.2
  • fftw-2.1.5-mpi
  • fftw-2.1.5-mvapich
  • fftw-2.1.5
  • fluent-6.3.26
  • fluent-6.2.16
  • freetype-2.4.2
  • games00
  • gamess-mvapich
  • gams-22.6
  • gcc-4.5.1
  • git-1.7.0.5
  • gmp-4.3.2
  • gsl-1.14
  • hdf5-1.8.4-patch1
  • hpmpi-2.01.00-08
  • intel-3.2.2.013
  • intel-cc-10.1.018
  • intel-cce-10.1.018
  • intel-fc-10.1.018
  • intel-fce-10.1.018
  • intel-flexlm
  • intel-ldb-10.1.018
  • intel-ldbe-10.1.018
  • intel-ippe-5.3.080
  • intel-mkl-10.0.5.025
  • intel-tbb-2.1.012
  • knotscape_1.01
  • lammps-14Apr10
  • lammps-31May07
  • libelf-0.8.12
  • libpng-1.4.3
  • liggghts_1p0p3
  • maple14
  • mathematica-7.0
  • mathematica-5.2
  • matlab-r2010a-dcs
  • matlab-r2009b
  • matlab-r2007a
  • moe-2009.10
  • moe-2007.09
  • mpc-0.8.1
  • mpfr-2.4.2
  • mpich-1.2.7pl
  • mpich-1.2.7pl-intel
  • mpich2-1.2.1pl
  • mpich2-1.2.1pl-intel
  • mpich2-1.2.1pl-pgroup
  • mpich-nag90
  • mpichold
  • mvapich2-1.4.1
  • mvapich2-1.4.1-gcc
  • mvapich2-1.4.1-intel
  • mvapich2-1.4.1-pgroup
  • mvapich2-1.2pl
  • mvapich2-1.2pl-intel
  • mvapich2-1.2pl-pgroup
  • nag90
  • namd-2.6bl
  • ncl-5.0.0
  • ncl-5.0.0-intel
  • nwchem-6.0
  • openFOAM
  • openfoam-1.7.1
  • paraview-3.4.0-Linux-x86_64
  • perl5
  • petsc-2.3.3-p8
  • petsc-2.3.3-p3
  • pgcc-8.0.3
  • pgi-10.2
  • pixman-0.19.4
  • pizza-3May07
  • ppl-0.11
  • python-2.7
  • R-2.11.1
  • texlive
  • vmd-1.8.4
  • zthread-2.3.2

Compilers (2)

What are my choices of C++ compilers on joe again?

PACE provides licenses for many compiler options, either from campus site licenses or via recurring funding from the Institute.  We have various versions of GNU, Intel, Portland Group and NAG compilers installed, as well as builds of mpich, mpich2, and mvapich2.  Please see the software list for full details.

"How do I run C++ codes on the joe cluster?"

Here's a trivial example that runs through the basic process of compiling and submitting a serial job to the force cluster.  More information on using the Moab job scheduler can be found in our user guide: 

Job Submission

The Job Scheduler

 

[user@force ~]$ ls
hello.cpp  hello.pbs
 
[user@force ~]$ cat hello.cpp
#include  using namespace std;
int main(void) {
  cout << "hello, world";
  return 0;
}
 
[user@force ~]$ cat hello.pbs
#PBS -N Hello
#PBS -l nodes=1:ppn=1
#PBS -l walltime=5:00
#PBS -q force
#PBS -k oe
 
$HOME/hello
 
[user@force ~]$ g++ -o hello hello.cpp 
 
[user@force ~]$ ls
hello hello.cpp hello.pbs
 
[user@force ~]$ ./hello
hello, world
 
[user@force ~]$ msub hello.pbs
45796.repace.pace.gatech.edu
 
[user@force ~]$ cat ../Hello.o45796
hello, world