Tag: tech support

The January maintenance period has begun.
All clusters will be inaccessible until maintenance is over.

Hi folks,

Just a reminder of our upcoming maintenance activities next week. Please see my previous blog post here: http://blog.pace.gatech.edu/?p=5449 for details.

In addition to the items described in the previous post, we will also be fixing up some quotas on home and project directories for some users who have no quotas applied. Per policy, all users should have a 5GB quota in their home directory.  A preliminary look through our accounts indicates that only one or two users have no quota applied here, and are over the 5GB limit.  We will be in touch with those users shortly to address the issue.  Project directory quotas are at the sized at the discretion of the faculty.  For those users without a quota on their project directory, we will apply a quota that is sufficiently sized such that all users remain under their quota.  After the maintenance day, we will provide a report to faculty detailing the project directory usage of their users, and work with them to make any adjustments needed.  Remember, the project directory quotas are simply intended to prevent an accidental consumption of a space that would negatively impact the work of other users of that storage.

Related to the home & project quotas, I’d also like to give you a heads up of some upcoming adjustments to the scratch space quotas.  Current policy is a 10TB soft quota and a 20TB hard quota.  Given the space problems we’ve been having with the scratch, we will be adjusting this to a 5TB soft quota, and a 7 TB hard quota.  This change should only affect a small handful of users.  Given the close proximity to our maintenance next week, we will be making this change at the end of January, NOT next week.  This is an easy first step that we can take to start addressing the recent lack of space on scratch storage.  We are looking at a broad spectrum of other policy and technical changes, including changing retention times, improving our detection of “old” files, as well as increasing capacity.  If you have any suggestions for other adjustments to scratch policy, please feel free to let me know.  Please remember that the scratch space is intended for transient data – not as a long term place to keep things.

Finally, we will also be completing the upgrade of the remaining RHEL5 portions of the Atlas cluster to RHEL6.  Likewise, we will continue the migration of the Athena cluster from RHEL5 to RHEL6, leaving only a few nodes as RHEL5.

–Neil Bright

…back to regularly scheduled events.

Our next maintenance window is fast approaching.  We will continue the 2-day downtimes, with the next one occurring Tuesday, January 14 and Wednesday, January 15.  The list of major changes is small this time around, but impactful.

The largest change, affecting all clusters, is a major update to the Moab & Torque scheduling system that is used to schedule and manage your jobs.  The upgraded versions fix a number of long-standing problems and scaling issues with command-timeouts, stability, and processing large job-sets.

The testflight cluster has been updated, and is available to anyone that wishes to test their submission processes against the new upgraded versions.In many cases, the processes used to submit and query your jobs will remain the same. For some, a change in the way that you use the system — may be required.  You will still be able to accomplish the same things, but may need to use different commands to do it.

We have updated our usage documentation to include a simple transition guide here.

In addition to the guide, we have also written a FAQ, which can be viewed by running the command ‘jan2014-faq‘ after logging in.

Because of the version differences between the old software and the new software, we will unfortunately not be able to preserve any jobs that are still in a queued state once maintenance begins. If you have any queued jobs going into maintenance, then you will need to resubmit them after maintenance.

The fixes planned for January also include the following:

Infrastructure:

• Operating System upgrades to the server running scheduling software for the “shared” clusters.  This will bring it up to the same level as the other scheduler servers.
• Adjustments to scalability & performance parameters on our GPFS filesystem.

Optimus cluster:

• Optimus users will have access to a new queue: ‘optimus-force-6’, as well as access to the iw-shared-6 queue.

Gryphon cluster:

UPDATE: All clusters are up and ready for service.

At this time, all PACE-managed clusters are believed to be working.
You should be able to login to your clusters and submit and run jobs.

Any jobs that were running before the power outage have failed, so please resubmit them.

Please let us know immediately if anything is still broken.

PACE Team

What happened

At around 0810 Thursday morning, Rich lost its N6 feed, half of the feed powering the Rich building and the Rich chiller plant. This also caused multiple failures in the high voltage vault in the Rich back alley, so Rich also lost its other feed, N5. However, the N5 feed was still up in the chiller plant. Though the chillers still had power, as a precaution operators transferred cooling over to the campus loop. Rich office space was without power, but the machine rooms failed over to the generator and UPSes.

PACE systems were powered down gracefully to prevent a hard-shutdown that would make recovery more difficult.

Original Post

This morning (December 19), the Rich datacenter suffered a power loss.
We had to perform an emergency shutdown of all nodes.

As we receive new information we will update this blog and the pace-availability email list.

COMSOL 4.4 – Student and Research versions

COMSOL Multiphysics version 4.4 contains many new functions and additions to the COMSOL product suite. These Release Notes provide information regarding new functionality in existing products and an overview of new products.
See the COMSOL Release Notes for information on updates to this version of COMSOL.

Using the research version of COMSOL

#Load the research version of comsol 

$ module load comsol/4.4-research 

$ comsol ... 

#Use the matlab livelink 

$ module load matlab/r2013b 

$ comsol -mlroot ${MATLAB} 

Using the classroom/student version of COMSOL

#Load the classroom/student version of comsol 

$ module load comsol/4.4 

$ comsol ... 

#Use the matlab livelink 

$ module load matlab/r2013b 

$ comsol -mlroot ${MATLAB} 

Additionally, we were able to complete a couple of bonus objectives.

The time has again come to discuss our upcoming quarterly maintenance.  As you may recall, our activities lasted well into night during our last maintenance in July.  Since then, I’ve been talking with various stakeholders and the HPC Faculty Governance Committee.  Starting with our October maintenance and going forward, we will be extending our quarterly maintenance periods from one to two days.  In October, this will be Tuesday the 15th & Wednesday 16th.  I’ve updated our schedule on the PACE web page.  Upcoming maintenance periods for January, April and July will be posted shortly.

Please continue reading below, there are a couple of things that will require user actions after maintenance day.

Scheduled fixes for October include the following:

• Project storage: We will deploy fixes for the remaining storage servers.  This will complete the roll out of these fixes initiated during our last maintenance period.  These fixes incorporate our best known stable Solaris platform at this point.  Between these fixes, and the networking fixes below, we believe this to resolve most, if not all, of the storage issues we’ve been having lately.
• Networking updates:  We have three categories of work here.  The first is to upgrade the firmware on all of the switches in our gigabit ethernet fabric.  This should solve the switch rebooting problem.  The second item is to finish the ethernet redundancy work we didn’t complete in July.  While this redundancy work will not ensure that individual compute nodes won’t suffer an ethernet failure, it does nearly eliminate the number of single points of failure in the network itself.  No user visible impact is expected.  We’re also planning on updating the firmware on some of our smaller Infiniband switches to bring them in line with the version of software we’re running elsewhere.
• Moab/Torque job scheduler: In order to mitigate some response issues with the scheduler, we are transitioning from a centralized scheduler server that controls all clusters (well, almost all) to a set of servers.  Sharing clusters will remain on the old server, and all of the dedicated clusters [1] will be distributed out to a series of new schedulers.  In all instances, we will still run the same _version_ of the software, but we’ll just have a fair bit more aggregate horsepower for scheduling.  This should provide a number of benefits, primarily in the response you see when submitting new jobs and querying the status of queued or running jobs.  Provided this phase goes well, we will look to upgrade the version of the moab/torque software in January.
  • There are some actions needed from the user community.  Users of dedicated clusters will need to resubmit jobs that did not get started before the maintenance.  The scheduler will ensure that it does not schedule a job that would not complete before maintenance, so this will only affect jobs that were submitted but never started.  You are affected if you _do not_ have access to the iw-shared-6 queue.
• New storage platform:  We have been enabling access to the DDN storage platform via the GPFS filesystem on all RHEL6 clusters.  This is now complete and we are opening up the DDN for investment.  Faculty may purchase drives on this platform to expand project spaces.  Please contact me directly if you are interested in a storage purchase.  Our maintenance activities will include an update to the GPFS software which provide finer grained options for user quotas.
• Filesystem balancing:  We will be moving the /nv/pf2 project filesystem for the FoRCE cluster to a different server.  This will allow some room for expansion and guard against it filling completely.  We expect no user-visiable changes here either, all data will be preserved, no paths will change, etc.  The data will just reside on a different physical server(s).
• vmWare improvements: We will be rebalancing the storage used by our virtual machine infrastructure (i.e. head nodes), and other related tasks aimed at improving performance and stability for these machines.  This is still an active are of preparation, so the full set of fixes and improvements remain to be fully tested.
• Cluster upgrades:  We will be upgrading the Atlantis cluster from RHEL5 to RHEL6.  Also, we will be upgrading 32 infiniband-connected nodes in Atlas-5 to Atlas-6.

[1] Specifically, jobs submitted to the following queues will be affected:

• aryabhata, aryabhata-6
• ase1-6
• athena, athena-6, athena-8core, athena-debug
• atlantis
• atlas-6, atlas-ge, atlas-ib
• complexity
• cssbsg
• epictetus
• granulous
• joe-6, joe-6-vasp, joe-fast, joe-test
• kian
• martini
• microcluster
• monkeys, monkeys_gpu
• mps
• optimus
• rozell
• skadi
• uranus-6

This is not a repeat from yesterday. Well, it is, just a different server 🙂

UPDATE 2013-08-08 2:23pm

/pb1 is now online, and should not fall over under heavy loads any more.

Have at it folks. Sorry it has taken this long to get to the final
resolution of this problem.

—- Earlier Post —-
Bad news:

If you haven’t been able to tell, the /pb1 filesystem has failed again.

Good news:

We’ve been working on a new load for the OS for all storage boxes
which we had hoped to get out on last maintenance day (July 17), but
ran out of time to verify whether it was

• deployable
• resolved the actual issue

Memo (Mehmet Belgin) greatly assisted me is testing this issue by finding some of the cases we’ve known to cause failures and replicating them against our test installs. Many loads were broken confirming our suspicions, and also confirming our new image. It will take heavy loads a LOT better than before.

With verification done, we have been planning to have all Solaris based storage switched to this by the end
of the next maintenance day (October 15).

However, due to need, this will be going on the PB1 fileserver is just a little bit. We’ve
verified the process of how to do this without impacting any data
stored on the server, so we anticipate having this fileserver back up
and running at 2:30PM, and the bugs which have been causing this
problem since April will have been removed.

I’ll follow up with progress messages.

UPDATE: 2013-08-07, 13:34 –

BEST NEWS OF ALL: /pc1 is now online, and should not fall over under heavy loads anymore.

Have at it folks. Sorry it has taken this long to get to the final
resolution of this problem.

Earlier Status:
Bad news:

If you haven’t been able to tell, the /pc1 filesystem has failed again.

Good news:

We’ve been working on a new load for the OS for all storage boxes
which we had hoped to get out on last maintenance day (July 17), but
ran out of time to verify whether it was

• deployable
• resolved the actual issue

Memo (Mehmet Belgin) greatly assisted me is testing this issue by finding some of the cases we’ve known to cause failures and replicating them against our test installs. Many loads were broken confirming our suspicions, and also confirming our new image. It will take heavy loads a LOT better than before.

With verification done, we have been planning to have all Solaris based storage switched to this by the end
of the next maintenance day (October 15).

However, due to need, this will be going on the PC1 fileserver is just a little bit. We’ve
verified the process of how to do this without impacting any data
stored on the server, so we anticipate having this fileserver back up
and running at 1:30pm, and the bugs which have been causing this
problem since April will have been removed.

I’ll follow up with progress messages.

Head nodes to many PACE clusters are currently down due to problems with our virtual machines.  This should not affect running jobs, but users are unable to login.  PACE staff are actively working to restore services as soon as possible.

The head nodes affected are:

• apurimac-6 – BACK ONLINE 2013/08/03 00:30
• aryabhata-6 – BACK ONLINE 2013/08/03 00:30
• ase1-6 – BACK ONLINE 2013/08/03 03:10
• athena – BACK ONLINE 2013/08/03 04:45
• atlantis – BACK ONLINE 2013/08/03 04:45
• atlas-6 – BACK ONLINE 2013/08/03 00:40
• cee – BACK ONLINE 2013/08/03 01:40
• chemprot – BACK ONLINE 2013/08/03 01:40
• complexity – BACK ONLINE 2013/08/03 01:40
• critcel – BACK ONLINE 2013/08/03 02:00
• ece – BACK ONLINE 2013/08/03 02:00
• emory-6 – BACK ONLINE 2013/08/03 02:20
• faceoff – BACK ONLINE 2013/08/03 03:10
• granulous – BACK ONLINE 2013/08/03 03:10
• isabella – BACK ONLINE 2013/08/03 03:10
• kian – BACK ONLINE 2013/08/03 03:10
• math – BACK ONLINE 2013/08/03 03:10
• megatron – BACK ONLINE 2013/08/03 03:10
• microcluster – BACK ONLINE 2013/08/03 03:10
• optimus-6 – BACK ONLINE 2013/08/03 00:30
• testflight-6 – BACK ONLINE 2013/08/03 03:10
• uranus-6 – BACK ONLINE 2013/08/03 00:30

The following nodes will likely generate SSH key errors upon
connection, as the key saving processes had not run on them. Please
edit your ~/.ssh/known_hosts file (Linux/Mac/Unix) and remove any host
entry with these names and save the new keys.

• ase1-6
• chemprot
• faceoff
• kian
• microcluster

Additionally, the following user-facing web services are also offline:

• galaxy