Author: ssarajlic3

CDA Lecture: Python and the Future of Data Analysis

Speaker:         Peter Wang, co-Founder of Continuum Analytics

Date:                 Friday, October 18, 2013

Location:        Klaus 1447

Time:                 2-3pm

 

Abstract:
While Python has been a popular and powerful language for scientific computing for a while now, its future in the broader data analytics realm is less clear, especially as market forces and technological innovation are rapidly transforming the field.

In this talk, Peter will introduce some new perspectives on “Big Data” and the evolution of programming languages, and present his thesis that Python has a central role to play in the future of not just scientific computing, but in analytics and even computing in general. As part of the discussion, many new libraries, tools, and technologies will be discussed (both Python and non-Python), both to understand why they exist and where they are driving technical evolution.

Bio:
Peter holds a B.A. in Physics from Cornell University and has been developing applications professionally using Python since 2001. Before co-founding Continuum Analytics in 2011, Peter spent seven years at Enthought designing and developing applications for a variety of companies, including investment bankers, high-frequency trading firms, oil companies, and others. In 2007, Peter was named Director of Technical Architecture and served as client liaison on high-profile projects. Peter also developed Chaco, an open-source, Python-based toolkit for interactive data visualization. Peter’s roles at Continuum Analytics include product design and development, software management, business strategy, and training.

COMSOL Workshop in Atlanta (9/10)

Here’a note from Siva Hariharan, COMSOL Inc., which we thought you might be interested in:

You’re invited to a free workshop focusing on the simulation capabilities of COMSOL Multiphysics. Two identical workshops will take place on Tuesday, September 10th in Atlanta, GA. There will be one AM session and one PM session. All attendees will receive a free two-week trial of the software.

During the workshop you will:

– Learn the fundamental modeling steps in COMSOL Multiphysics

– Experience a live multiphysics simulation example

– Set up and solve a simulation through a hands-on exercise

– Learn about the capabilities of COMSOL within your application area

 

Programs:

AM Session

9:30am – 10:45am An Overview of the Software

10:45am – 11:00am Coffee Break

11:00am – 12:30pm Hands-on Tutorial

 

PM Session

1:30pm – 2:45pm An Overview of the Software

2:45pm – 3:00pm Coffee Break

3:00pm – 4:30pm Hands-on Tutorial

 

Event details and registration: http://comsol.com/c/tt1

 

Seating is limited, so advance registration is recommended. 

Feel free to contact me with any questions.

 

Best regards,

Siva Hariharan

COMSOL, Inc.
1 New England Executive Park
Suite 350
Burlington, MA 01803
781-273-3322
siva@comsol.com

PB1 bad news, good news

This is not a repeat from yesterday. Well, it is, just a different server 🙂

UPDATE 2013-08-08 2:23pm

/pb1 is now online, and should not fall over under heavy loads any more.

Have at it folks. Sorry it has taken this long to get to the final
resolution of this problem.

—- Earlier Post —-
Bad news:

If you haven’t been able to tell, the /pb1 filesystem has failed again.

Good news:

We’ve been working on a new load for the OS for all storage boxes
which we had hoped to get out on last maintenance day (July 17), but
ran out of time to verify whether it was

  • deployable
  • resolved the actual issue

Memo (Mehmet Belgin) greatly assisted me is testing this issue by finding some of the cases we’ve known to cause failures and replicating them against our test installs. Many loads were broken confirming our suspicions, and also confirming our new image. It will take heavy loads a LOT better than before.

With verification done, we have been planning to have all Solaris based storage switched to this by the end
of the next maintenance day (October 15).

However, due to need, this will be going on the PB1 fileserver is just a little bit. We’ve
verified the process of how to do this without impacting any data
stored on the server, so we anticipate having this fileserver back up
and running at 2:30PM, and the bugs which have been causing this
problem since April will have been removed.

I’ll follow up with progress messages.

PC1 bad news, good news

UPDATE: 2013-08-07, 13:34 –

BEST NEWS OF ALL: /pc1 is now online, and should not fall over under heavy loads anymore.

Have at it folks. Sorry it has taken this long to get to the final
resolution of this problem.

Earlier Status:
Bad news:

If you haven’t been able to tell, the /pc1 filesystem has failed again.

Good news:

We’ve been working on a new load for the OS for all storage boxes
which we had hoped to get out on last maintenance day (July 17), but
ran out of time to verify whether it was

  • deployable
  • resolved the actual issue

Memo (Mehmet Belgin) greatly assisted me is testing this issue by finding some of the cases we’ve known to cause failures and replicating them against our test installs. Many loads were broken confirming our suspicions, and also confirming our new image. It will take heavy loads a LOT better than before.

With verification done, we have been planning to have all Solaris based storage switched to this by the end
of the next maintenance day (October 15).

However, due to need, this will be going on the PC1 fileserver is just a little bit. We’ve
verified the process of how to do this without impacting any data
stored on the server, so we anticipate having this fileserver back up
and running at 1:30pm, and the bugs which have been causing this
problem since April will have been removed.

I’ll follow up with progress messages.

Login Node Storage Server Problems

Last night (2013/06/30), one of the storage servers that is responsible for many of the cluster login nodes encountered some major problems.
These issues are preventing the login nodes from allowing any user to login or use the server.
Following is a list of the affected login nodes:
cee
chemprot
cns
cygnus-6
force-6
force
math
mokeys
optimus
testflight-6

We are aware of the problem and we are working as quickly as possible to fix this.
Please let us know of any problems you are having that may be related to this.
We will keep you posted about our progress.

Intel Cluster Studio 2013 XE Installed

The Intel Cluster Studio 2013 XE software suite installation adds several new and useful tools for PACE users.

  • VTune: Intel® VTune™ Amplifier XE 2013 is a serial and parallel performance profiler for C, C++, C#, Fortran, Assembly and Java.
  • Inspector: Intel® Inspector XE is an easy to use memory debugger and thread debugger for serial and parallel applications.
  • Advisor: Intel® Advisor XE is a threading prototyping tool for C, C++, C# and Fortran.

This installation includes updated versions of many currently installed packages. The updates include:

  • MKL – updated to 11.0.1
  • TBB – updated to 4.1
  • IPP – updated to 7.1.1
  • Compilers (C, C++, Fortran) – updated to 13.2.146

To use the new or updated software, please load whichever modules are appropriate:

  • intel/13.2.146 (loads the C, C++, and Fortran compilers)
  • vtune/2013xe (loads VTune)
  • advisor/2013xe (loads Advisor)
  • inspector/2013xe (loads Inspector)
  • tbb/4.1 (loads the Thread Building Blocks)
  • ipp/7.1.1 (loads the Performance Primitives)
  • mkl/11.0.1 (load the Math Kernel Library)

For information on using VTune, Inspector, Advisor, or any of the Intel tools, see the Intel Cluster Studio XE site.

New 128-procs Allinea DDT license on PACE clusters

Allinea DDT is a powerful parallel debugger with an easy-to-use GUI. You can run it by loading its module (module load ddt/3.2) and entering “ddt”. Some introduction level information can be found in “https://pace.gatech.edu/workshop/DebuggingProfiling.pdf“.

We extended our single-user 32-procs license to multi-user 128 procs. Aside from the increased number of processors, this license allows multiple users to use the software at the same time, as long as the total number of processors do not exceed 128. E.g., two users can use the software with 64procs run each.

Happy debugging!

 

 

PACE Systems Back Online

The fileserver has recovered, and all headnodes are now accessible. The jobs running off scratch should continue from where they left. You have access to all files, including the scratch. The server is still performing reconstruction of data, which may slow down the system (especially on volumes v0 and v3) for a few more hours. This slowness will go away when the reconstruction is complete.

We are expecting to receive the failed part tomorrow (6/6). The fileserver can function without this part and its installation will not cause any interruptions.

Once again, thank you for bearing with us while we were working on this problem. If you have jobs that you think crashed due to this problem, please send us an email at pace-support@oit.gatech.edu.

Login Problems, current situation

The Panasas fileserver (scratch storage) crashed today while recovering from a hardware problem. This causes the headnodes (that mount Panasas) to hang, and they are not accessible via SSH now.

We do have a way to disable Panasas and give you access to headnodes right away, without the panasas storage. However, doing so will crash all of the jobs using the scratch space. We do not want that, especially considering that some jobs have been running for days.

We are now running a filesystem check on the system, which will take 3 to 4 hours. This is required to prevent data corruption. After this process, Panasas should recover and the jobs will continue running. At the point, the headnodes will become accessible again.

If you urgently need to access your data in your home or project directories, please contact us at pace-support@oit.gatech.edu. We might be able to help you access your files via a headnode that does not mount Panasas.

The filesystem check has been running for 40 minutes and current at 26% (by 12:25pm EST).

Thank you once again for your understanding and patience, and we apologize for this inconvenience,