We received multiple reports of jobs getting killed around 2:00am. After further investigation, we have found the cause and made the corrections required to prevent this from happening again. Here’s a detailed explanation of what caused the job failures:

Each individual machine in PACE, including workstations in some cases, has an OS-software stack that is maintained by a single-sourced service called GTSWD. During maintenance periods we often use the GTSWD service to push-out new OS updates, firmware updates, and system service updates.

One of the updates we pushed out during the last maintenance window was a new panfs client, which is responsible for mounting the /scratch filesystem. The process used to update the panfs client came in two stages:

#1 initiate the client installation under the assumption that the node was free (this was a valid assumption at that time because this was being done during the maintenance window).

#2 replace the process that did the client installation with another update process that would first check to see if the change had already been made, but more importantly, would not make the assumption that the node was free.

On some of the compute nodes, #2 didn’t not get applied, and so every day at 2am since maintenance day, process #1 has been attempting to umount and remount /scratch, causing the failure of several user jobs. The reason #2 did not get applied, is because the GTSWD service-source is now completely overwhelmed by the number of PACE nodes that we now have trying to use it, and had not been able to automatically apply the updates to a small subset of our nodes.

In terms of this particular problem, we have manually updated all the RHEL5 nodes that were still using process #1; that will stop any more jobs that use RHEL5 queues from getting killed at 2am. In terms of the capacity problem, we’re going to be adding more capacity to the GTSWD service so that it can support all of the PACE nodes. On the RHEL6 nodes we’ve updated the execution instruction sets on our distribution system such that, when updated, they will not attempt to use process #1.

We are sorry for the time this has cost you, and also for not correcting the problem sooner. The failure rate of GTSWD has been historically very low, and thus is usually one of the very last things we look at when trying to determine the source of a problem.

 

Program Name: Major Research Instrumentation (MRI) Program

Program Website: http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=5260

A copy of this solicitation is available at: http://www.nsf.gov/pubs/2011/nsf11503/nsf11503.htm

Purpose of the Program: the MRI program assists with the acquisition or development of shared research instrumentation that is, in general, too costly and/or not appropriate for support through other NSF programs. Instruments are expected to be operational for regular research use by the end of the award period. For the purposes of the MRI program, proposals must be for either acquisition or development of a single instrument or for equipment that, when combined, serves as an integrated research instrument (physical or virtual).  The MRI program does not support the acquisition or development of a suite of instruments to outfit research laboratories/facilities or to conduct independent research activities simultaneously.  Further guidance on appropriate requests can be found in the MRI Frequently Asked Questions (FAQs) at http://www.nsf.gov/od/oia/programs/mri.

An institution may be either a lead or a sub in up to 3 proposals. Since Georgia Tech anticipates having 3 lead proposals, proposals as subs will only be considered should we fall short of the 3 leads. Deadline for submission to NSF is January 24, 2013 by 5:00 p.m. EST.  However, the internal deadline is Friday, November 9, 2012.

Internal Process

If you would like to submit an MRI proposal on behalf of Georgia Tech, please submit a 4-page white paper via T-Square by November 9, 2012.  Submission instructions:

1. Log on to http://www.tsquare.gatech.edu

2. Under the “My Workspace” tab, click on the “Membership” sidebar.

3. Click on “Joinable Sites” and join “MRI Submissions.”

4. Submit your paperwork as one pdf using the Drop Box tool.

5. Direct any questions to Gail Spatt: 404.385.8334 or spatt@gatech.edu

Format

Please use the following format for your white papers:

Pages 1-3

• ACQUISITION or DEVELOPMENT (clearly indicate)
• Project Title
• Principle Investigator(s) and Project Director(s)
• Project Description to include: research activities enabled; description of the research instrumentation and needs; impact on research and training infrastructure; and management plan.

Page 4

• Budget and justification
• Cost-sharing – note that cost-sharing at 30% is required for the MRI program; contributions may be from any non-Federal source, including non-Federal grants/contracts, and may be in cash or in-kind (see Section V. B. Budgetary Information of the Program Solicitation for full details).

As always, this information is also available at this website: http://www.evpr.gatech.edu/faculty-resources-and-funding-opportunities/major-research-instrumentation-program

 

We have completed our maintenance activities.  Head nodes are online again and queued up jobs are being released.

Our filesystem correction activities on the scratch found eight “objects” on the v7 volume to be damaged and were automatically removed.  Unfortunately, the process provides no indication which file or directory was problematic.

As always, please followup to pace-support@oit.gatech.edu with any problems you may see, ideally with the pace-support.sh script discussed here: https://pace.gatech.edu/support.

PACE Maintenance Day is underway.
All compute nodes are off, and all login nodes should be inaccessible.

The Network team will be performing some scheduled maintenance this Saturday morning.  This may impact connectivity between your workstations/laptops/home, but should not affect  jobs running within PACE.  However, if your job requires access to network services outside of the PACE cluster (e.g. a remote license server), this maintenance may affect your jobs.

For further information please see the maintenance announcement on status.oit.gatech.edu.

Dear PACE Users,

We have a new tool to announce. If you would like to check the status of any PACE queue, you can now run:

pace-check-queue <queuename>

substituting the queuename with  the name of the queue you would like to check. This tool has a column, which tells you whether a node is accepting jobs or not, including a human readable explanation. This tool provides, at one glance, the following information:

* Which nodes are included in the queue

* Which nodes accept jobs and which don’t (and if they don’t, why)

* How may cores and how much memory each node has, and what percent of them are being used

* Overall usage (CPU/Memory) levels for the entire queue.

(This information is refreshed every half an hour)

We had recently announced a new tool, pace-stat, to check the status of your queues. These tools complement each other, so feel free to use both. Please report any down/problem nodes that you see in the list to pace-support@oit.gatech.edu.

Hope these new tools will provide you with a better HPC environment. Happy computing!

PS: These tools are continuously being developed, therefore your feedback and suggestions for improvements are always welcome!

It’s that time again.  We’ve been working with our scratch storage vendor (Panasas) quite a lot lately, and think we finally have some good news.  Addressing the scratch space will be a major thrust of this quarterly maintenance, and we are cautiously optimistic that we will see improvements.  We will also be applying some VMware tuning to our RHEL5 virtual machines that should increase responsiveness of those head nodes & servers.  Completing upgrades to RHEL6 for a few clusters and a few other minor items round out our activities for the day.

Scratch storage

We have been testing new firmware on our loaner Panasas storage.  Despite our best efforts, we have been unable to replicate our current set of problems after upgrading our loaner equipment to this firmware.  This is good news!  However, simply upgrading is insufficient to fully resolve our issues.  So on maintenance day, we will be performing a number of tasks related to the Panasas.  After the firmware update, we need to perform some basic file integrity checks – the equivalent of a UNIX fsck – on a copule of volumes.  This process requires those volumes to be offline for the duration.  After this, we need to perform reads of every file on the scratch that was created before the firmware upgrade.  Based on our calculations, this will take weeks.  Fortunately, this process can happen in the background, and with the filesystems online and otherwise operating normally.  The net result is that the full impact of our maintenance day improvements to the scratch will not likely be realized for a couple of weeks.  If there are files (particularly large ones) that you no longer need and can delete, this process will go faster.  We will also be upgrading the Panasas client software on all compute nodes to (hopefully) address performance issues.

Finally, we will also be instituting a 20TB per user hard quota in addition to the 10TB per user soft quota currently in place.  Users that exceed the soft quota will receive warning emails, but writes will succeed.  Writes will fail for users that attempt to exceed the hard quota.

VMware tuning

With some assistance from the Architecture and Infrstructure directorate in OIT, we will be making a number of adjustments to our VMware world.  The most significant of which is adjusting the filesystem alignment of our RHEL5 virtual machines.  Users of RHEL5 head nodes are likely to see the most improvement.  We’ll also be installing the VMware tools packages and applying various tuning parameters enabled by this package.

RHEL6 upgrades

The remaining RHEL5 portions of the clusters below will be upgraded to RHEL6.  After maintenance day, RHEL5 will be unavailable to these clusters.

• Uranus
• BioCluster
• Cygnus

Misc items

• Configuration updates to redundant network switches serving some project storage
• Capacity expansion of the ECE file server
• Serial number updates to a small number of compute nodes lacking serial numbers in the BIOS
• Interoperability testing of Mellanox Infiniband switches
• Finish project directory migration of two remaining Optimus users

We have been able to bring /nv/pc5 back online, but at a cost to redundancy. One of the network interfaces/cables/switches is not behaving, but when we tried disconnecting various combinations of cables, we found one that caused the filesystem to be immediately available to all nodes.

Considering how close maintenance day is (10/16/12), spending time isolating the cable/switch/interface problem now only means more time for this filesystem to be offline as equipment gets retested. Waiting until maintenance day will cause the least disruption for Cygnus pc5 users who have their last run of jobs and take some time pressure off of us to make sure we have resolved the issue in its entirety before bringing all resources back online.

Despite the loss of redundancy, functionality is NOT affected. Only in the case of an additional switch or cable failure between now on October 16 will functionality be impacted.

It appears that we have an issue with the server housing the /nv/pc5 filesystem, which contains a subset of the Cygnus cluster users. We’re trying to isolate the source of the problem, but we have yet to actually find a pattern to why it is available on some nodes and not on others.

Around 8, 8:30pm on September 28, 2012, a power event took down the TSRB data center, knocking a significant fraction of the Joe cluster offline.

With assistance from Operations, we are now bringing these nodes online after determining that several of the management switches for these nodes did not recover from the event gracefully. As these switches control our ability to manage the nodes, we had to wait until the switches were available to bring nodes online, now at about 4pm on September 29, 2012.

Jobs that were running on these nodes (iw-a2-* and iw-a3-*) at the time of the outage may have terminated abnormally. Jobs scheduled but not running should be fine.

UPDATE @ 4:40pm, 2012-09-29: All nodes are online.