Posts

PACE quarterly maintenance – April ’15

Greetings everybody.  It’s again time for the quarterly PACE maintenance.  As usual, we will have all PACE clusters down Tuesday and Wednesday of next week, April 21 and 22.  We’ll get started at 6:00am on Tuesday, and have things back to you as soon as possible.There’s some significant changes this time around, so please continue on.

Moab/Torque scheduler:
Last maintenance period, we deployed a new scheduler for the Atlas and Joe clusters.  This time around, we’re continuing that rollout to the rest of our clusters.  Some of the highlights are:

  • increased responsiveness from commands like qsub & showq
  • need to resubmit jobs that haven’t run yet
  • removal of older, incompatible versions of mvapich and openmpi
  • required changes to our /usr/local software repository

Mehmet Belgin has posted a detailed note about the scheduler upgrade on our blog here.  He has also posted a similar note about the related updates to our software repository here.

Additionally, the x2200-6.3 queues on the Atlas cluster will be renamed to atlas-6-sunib and atlas-6-sunge.

Networking:
We’ve deployed upgraded network equipment to upgrade the core of the PACE network to 40-gigabit ethernet and will transition to this new core during the maintenance period.  This new network brings additional capability to utilize data center space outside of the Rich building, and provides a path for future 100-gigabit external connectivity and ScienceDMZ services.  Stay tuned for further developments. 😉  Additionally, the campus network team will be upgrading the firmware of a number of our existing switches with some security related fixes.

Storage:
The network upgrades above will allow us to relocate some of our project directory servers to OIT data center space on Marietta Street, as we’re pressed for generator-protected space in Rich.  We will also be doing some security patching, highly recommended updates and performance optimizations on the DDN/GPFS storage.  As a stretch goal, we will also migrate some filesystems to GPFS.  If we are pressed for time, they will move with their old servers. Details regarding which filesystems are available on our blog here.

Operating System patching:
Last, but not least, we have a couple of OS patches.  We’ll complete the rollout of a glibc patch for the highly publicized “Ghost” vulnerability, as well as deploy a bug fix for autofs that addresses a bug which would sometimes cause a failure to mount /nv filesystems.

Important Changes to PACE storage

During our quarterly maintenance period next week, we will relocate some of our project directory servers to OIT data center space on Marietta Street, as we’re pressed for generator-protected space in Rich.  This is a major undertaking, with over 20 servers moving.  Our intent is that no change is needed on your part, but wanted to ensure transparency in our activities.  The list below contains all of the affected filesystems.  The list of filesystems to which you have access can be obtained with the ‘pace-quota’ command.

  • home directories for all clusters except Gryphon and Tardis
  • /nv/pb4, /nv/archive-bio1 (BioCluster)
  • /nv/hchpro1, /nv/pchpro1 (Chemprot)
  • /nv/pas1 (Enterprise)
  • /nv/pase1 (Ase1)
  • /nv/pb2 (Optimus)
  • /nv/pbiobot1 (BioBot)
  • /nv/pc4, /nv/pc5, /nv/pc6 (Cygnus)
  • /nv/pccl2 (Gryphon, legacy)
  • /nv/pcoc1 (Monkeys)
  • /nv/pe1, /nv/pe2, /nv/pe3, /nv/pe4, /nv/pe5, /nv/pe6, /nv/pe7, /nv/pe8, /nv/pe9, /nv/pe10, /nv/pe11, /nv/pe12, /nv/pe13, /nv/pe14 (Atlas)
  • /nv/hp1, /nv/pf1, /nv/pf2 (FoRCE)
  • /nv/pface1 (Faceoff)
  • /nv/pg1 (Granulous)
  • /nv/pggate1 (GGate)
  • /nv/planns (Lanns)
  • /nv/pmart1 (Martini)
  • /nv/pmeg1 (Megatron)
  • /nv/pmicro1 (Microcluster)
  • /nv/pska1 (Skadi)
  • /nv/ptml1 (Tmlhpc)
  • /nv/py2 (Uranus)
  • /nv/pz2 (Athena)
  • /nv/pzo1, /nv/pzo2 (backups for Zohar and NeoZhoar)

Additionally, the following filesystems will be migrated to GPFS:

  • /nv/pcee1 (cee.pace)
  • /nv/pme1, /nv/pme2, /nv/pme3, /nv/pme4, /nv/pme5, /nv/pme6, /nv/pme7, /nv/pme8 (Prometheus)

As a stretch goal, we will also migrate the following filesystems to GPFS.  If we are pressed for time, they will move with their old servers as listed above.

  • /nv/hp1, /nv/pf1, /nv/pf2 (FoRCE)
  • /nv/pas1 (Enterprise)
  • /nv/pbiobot1 (BioBot)
  • /nv/pccl2 (Gryphon, legacy)
  • /nv/pggate1 (GGate)
  • /nv/planns (Lanns)
  • /nv/ptml1 (Tmlhpc)

Important Notes on Coming PACE Scheduler Upgrades

We have been running an upgraded scheduler version on Joe and Atlas clusters, who had graciously volunteered to test it out since January. This version brings significant performance and stability improvements, and we are looking forward to roll out the upgrades to the rest of the PACE universe during this maintenance period. Please note the following important changes, which will apply to all PACE users.

  • The new schedulers are not compatible with MPI versions older than mvapich/1.9 and openmpi/1.6.2. If you are using one of the older MPI stacks (a warning would be printed when you load their modules), you will need to replace it with one of the recent versions. This motivated the creation of  a new and improved software repository, which will be available after the maintenance day. For more details, please see our related post.

 

  • The current version uses a different type of database, so we will not be able to migrate submitted jobs.  The scheduler will start with an empty queue, and you will need to resubmit your jobs after the maintenance day. This applies to Joe and Atlas jobs as well, as we are merging exclusive queues on a new and more powerful server with the exception of Tardis, Gryphon and Testflight.

 

  • We will start using the “node packing” policy which allocates as many jobs on a node as possible before jumping on the next one. With the current version, users can submit many single-core jobs, each landing on a separate node, making it more difficult for the scheduler to start jobs that require entire nodes.

 

  • This version fixes a bug that prevents use of msub for interactive jobs. The recommendation from the vendor company is to use “qsub” for everything (we confirmed that it’s much faster than msub), but this bug fix gives you the freedom to pick either tool.

 

  • There will no longer be a discrepancy between job IDs generated by msub (Moab.###) and qsub (####). You will always see a single job ID (in plain number format) regardless of your msub/qsub preference.

 

  • Speed — new versions of Moab and Torque are now multithreaded, making it possible for some query commands (e.g. showq) to return instantly regardless of the load on the scheduler. Currently, when a user submits a large job array, these commands usually timeout.

 

  • Introduction of cpusets — when a user is given X cores, he/she will not be able to use more than that. Currently, users can easily violate the requested limits by spawning any number of processes/threads and Torque cannot do much to stop that. The use of cpusets will significantly reduce the job interference and allows us to finally use ‘node packing’ as explained above.

 

  • Several other benefits from bug fixes and improvements are (including but not limited to) less number of zombie processes, lost output files, missing array jobs. We also expect visible improvements in job allocation times and less frequent command timeouts.

 

We hope these improvements will provide you with a more efficient and productive computing environment. Please let us know (pace-support@oit.gatech.edu) if you have any concerns or questions!

Important Changes to PACE Scientific Software Repository

As announced earlier, we will remove a set of old MPI stacks (and applications that use them) from the PACE software repository after the April maintenance day. This is required by the planned upgrade of the schedulers (torque and moab), which use libraries that are incompatible with the old MPI stacks. Some MPI-related Python modules (e.g. mpi4py) are built on one of these old MPI versions (namely mvapich2/1.6) and they will also stop working with the new scheduler.

Old MPI versions are also known to have significant performance and scalability problems, and they are no longer supported by developers, therefore their expulsion was inevitable regardless of the scheduler upgrades. Namely, all versions older than than “mvapich2/1.9” and “openmpi/1.6.2” are known to be incompatible, and will be removed along with applications that are compiled with them. MPI stacks newer than these versions are compatible with the new scheduler version, so they will continue to be available. PACE team is ready to offer assistance with all the changes you may need to replace these old MPI versions with new versions with minimal interruptions to your research.

We saw these problems as an opportunity to start creating a new and improved software repository almost from scratch, which not only fixes the MPI problems, but also provides added benefits such as:

* a cleaner MPI versioning without long confusing subversions such as “1.9rc1” or “2.0ga”: You will see a only a single subversion for each major release, e.g.,

mvapich2: 1.9, 2.0, 2.1, …
openmpi: 1.6, 1.7, 1.8, …

* latest software versions: We showed a best effort to compile the most recent (stable) versions as we could, unless they had compilation problems or proved to be buggy.

* a new python that allows parallelization without requiring InfiniBand (IB) network: Current python uses mvapich2, which requires IB network. The new python, on the other hand, will employ openmpi, which can run on *any* node regardless of their network connection while still taking advantage of IB when available.

We will start offering this new repository as an alternative after the April maintenance day. Switching between old and the new repository will be as easy as loading/unloading a module named “newrepo”. E.g.:

# Make sure there are no loaded modules
$module purge
$module load newrepo

… You are now using the new repo …

# since newrepo is also a module itself, ‘module purge’ will put you back in the old repo
$module purge

… You are back in the old repo …

The current plan is to decommission the old repository after the July maintenance, therefore strongly encourage you to try the new repository (which is still beta) as soon as possible to ensure a smooth transition. If the new repository is working for you, continue to use it and never look back. If you notice problems or missing components, you can continue to use the old repository while we are working on fixing them.

Please keep in mind that the new repo is created almost from scratch, so expect changes in module names, as well as new set of dependencies/conflicts between the modules. PACE team is always ready to provide module suggestions for your applications, or answer any other questions that you may have.

We hope the new repository will make a positive contribution to your research environment with visible improvements in performance, stability and scalability.

Thanks!
PACE Team

Python users of PACE: Meet “Anaconda”

We are happy to announce the availability of “Anaconda” python distribution for data analytics and scientific computing. This distribution is a commercial product, which is free for academic use.

https://store.continuum.io/cshop/anaconda/

The default pyhton (2.7) on PACE systems already offers a comprehensive list of scientific software, so here’s a quick pros & cons to help you decide when to use anaconda:

pros of Anaconda
—————
1. Standalone distribution with no particular dependency on a compiler or MPI stack.
2. A long list of supported libraries for python 2.7 and python 3.4 alike (PACE support for python 3.4 is little to none)
3. Fully tested and supported distribution, backed by the creators of numpy and scipy.
4. Very easy to add and upgrade packages using the “conda” package manager. PACE will regularly upgrade all packages to their latest versions.
5. Constitutes an alternative to PACE distribution when some libraries are found to be missing, outdated, or buggy.

cons of Anaconda
—————
1. Lacks MKL optimizations (which requires a separate license). PACE compiled libraries usually come with MKL and may outperform Anaconda, particularly for linear algebra routines and fast Fourier transform.
2. Limited to the libraries supported by the “conda” package manager (although the list is pretty comprehensive). In comparison, PACE distribution can be extended with any given compatible modules and libraries.
3. Cannot be substituted with PACE versions for scientific packages that are compiled with PACE python.

should you decide to give it a try, here’s how to use it:

# Make sure you remove all PACE python module(s)
module rm python        # Or better yet: module purge

# For python 2.7:
module load anaconda2/2.1.0

# For python 3.4:
module load anaconda3/2.1.0

then use “python” as usual.

Please let us know if you find that a library you need is provided by Anaconda but not PACE python, so we can add it. PACE python will continue to be the primary python distribution, hence must continue to offer all of the libraries that you need for your research.

As always, please contact us at pace-support@oit.gatech.edu and let us know if you have any problems. You can also contact mehmet.belgin@oit.gatech.edu directly to leave general feedback about your experience with this distribution.

Happy computing!

International HPC Summer School 2015

Graduate students and postdoctoral scholars from institutions in Canada, Europe, Japan and the United States are invited to apply for the sixth International Summer School on HPC Challenges in Computational Sciences, to be held June 21-26, 2015, in Toronto, Canada. Applications are due March 11, 2015. The summer school is sponsored by Compute/Calcul Canada, the Extreme Science and Engineering Discovery Environment (XSEDE) with funds from the U.S. National Science Foundation, the Partnership for Advanced Computing in Europe (PRACE) and the RIKEN Advanced Institute for Computational Science (RIKEN AICS).

Leading Canadian, European, Japanese and American computational scientists and HPC technologists will offer instruction on a variety of topics, including:

HPC challenges by discipline (e.g, earth, life and materials sciences, physics)
HPC Programming Proficiencies
Performance analysis and profiling
Algorithmic approaches and numerical libraries
Data-intensive computing
Scientific visualization
Canadian, EU, Japanese and U.S. HPC-infrastructures

The expense-paid program will benefit advanced scholars from Canadian, European, Japanese and U.S. institutions who use HPC to conduct research. Interested students should apply by March 11, 2015. Meals, housing, and travel will be covered for the selected participants. Applications from graduate students and postdocs in all science and engineering fields are welcome. Preference will be given to applicants with parallel programming experience, and a research plan that will benefit from the utilization of high performance computing systems.

More details available at: https://ihpcss2015.computecanada.ca/wp/

Free Matlab Seminars Feb 5 and 6

MATLAB and Data Analytics Seminars at Georgia Tech

Mathworks-hosted complimentary MATLAB and Data Analytics seminars. Faculty, staff, researchers and students are all welcome to attend.

Session 1
Thursday, February 5: 2:00 p.m. – 4:00 p.m
Location: Student Success Center, Press Room A/B
Data Analysis and Visualization in MATLAB

Learn how MATLAB can be used to visualize and analyze data, perform numerical computations, and develop algorithms.

Session 2
Friday, February 6: 2:00 p.m. – 4:00 p.m
Location: Klaus 1116
Data Science/Data Analytics and Scaling to Big Data

Using Data Analytics to turn large volumes of complex data into actionable information can help you improve design and decision-making processes. However, developing effective analytics and integrating them into business systems can be challenging. In this seminar you will learn approaches and techniques available in MATLAB® to tackle these challenges

To view complete session descriptions and register, visit:
https://go2.mathworks.com/georgia-institute-of-technology-sem-na-65703

Temporary disruption for many home directories

The server housing about half of the PACE affiliated home directories lost of of its root drives (ie: the operating system, not data drives) and that threw the system into a state where it was effectively only processing the intterrupts from that failure. We’ve rectified the situation so that the server can provide access to files again.

Affected:

hbiobot1     biobot
hcee2          cee
hcfm1          isabella
hface1         faceoff
hggate1      ggate
hmeg1        megatron
hp10           biocluster
hp12           aryabhata
hp14           atlantis
hp16           force
hp18           optimus
hp20          ece
hp22          prometheus
hp24          math
hp26          cee
hp28          granulous
hp8            athena
hpampa1  pampa
html1        tmlhpc

Short duration of unavailability (Feb 3, 2015)

On Feburary 3, at 7:00am, in coordination with our Network Team:

– Network team will have one of the main PACE firewall appliances rebooted in order to fix some isses we have been having with these appliances. This may cause a disruption in networking from outside of PACE in to the head nodes of all PACE clusters.

– PACE will also reboot all of the head nodes for PACE clusters to enact patches to fix the GHOST vulnerability.

These disruptions will result in a 15 minutes period of unavailability for the head nodes. Any login sessions and data transfers into PACE established prior to that time will be terminated, and any processes running on the head nodes themselves will also be terminated. Submitted and running user jobs, however, will remain unaffected, as this will only affect the connections from outside of PACE to inside — all internal operations will function as normal.