How to use job-specific local temporary directories

Problem:

Every Linux node comes with a "/tmp" directory that's local to the node. Since this is a local disk mounted locally on the node, it's faster than the network attached storage that PACE provides (home, data, scratch...). For this reason, a common practice is to use this location temporarily and delete the files before the job completes running. 

However, job/scripts that don't have a proper /tmp cleanup step have been creating a lot of problems. Due to the limited local disk space (<20GB on most nodes), even a single job would be enough to fill up the available space, causing the node to go offline.

Solution:

As of May, 2018 PACE schedulers are configured to automate this process, without relying on user action. First, each node now comes with a "/scratch" directory, which is created on a local disk. Please note that this is different than the "~/scratch" directory found in your home directories, which is network attached. And here's the good part: this directory will be automatically deleted by the scheduler when the job is complete, without requiring any action by the user. 

Each new job will automatically create a directory under /scratch, named after the job ID, e.g.:

/scratch/20986925.shared-sched.pace.gatech.edu

And the ${TMPDIR} environment variable (which normally points to '/tmp') is assigned to this path, instead of /tmp:

$ cd ${TMPDIR}
$ pwd
/scratch/20986925.shared-sched.pace.gatech.edu

At this point, applications that honor the standard "${TMPDIR}" variable will start using this location instead of /tmp. More importantly, users can substitute all occurrences of "/tmp" with "${TMPDIR}" in their codes and scripts to ensure that they'll use this location instead. 

After the job is complete, the scheduler will delete this directory with all of the files in it, without needing a manual cleanup step.

Required User Action:

Using this mechanism is critical to prevent large number of nodes running out of local space. For this reason, PACE requires all users to use this ${TMPDIR} variable in their workflow instead of /tmp. In other words, if you have paths specified in your PBS scripts or codes, please replace all occurences of "/tmp" with "${TMPDIR}", e.g:

If you have a line like this:

python my_python_code.py > /tmp/intermediate_results.txt

Replace as follows:

python my_python_code.py > /${TMPDIR}/intermediate_results.txt

Please contact pace support if you have any questions or need assistance with changing your workflow accordingly. 

 

How to ensure local storage availability when submitting jobs?

PACE clusters consist of many nodes with different specifications, including different local drive capacities ranging from 20G to 7TB.

The good news is that the scheduler comes with a mechanism to track the available local temporary directory. For example, if you need 10GB space for your job, you can request it easily as follows:

#PBS -l file=10g

This will make sure that your job will be allocated on a node that has at least 10gb free on ${TMPDIR}.