The Performance API (PAPI) and TAU are two of the most common open source profiling tools, and they are now available for PACE users, including support for hardware counters and threading.
PAPI description, from their website:
The PAPI project specifies a standard application programming interface (API) for accessing hardware performance counters available on most modern microprocessors. PAPI provides two interfaces to the underlying counter hardware; a simple, high level interface for the acquisition of simple measurements and a fully programmable, low level interface directed towards users with more sophisticated needs.
TAU description, from their website:
TAU Performance System is a portable profiling and tracing toolkit for performance analysis of parallel programs written in Fortran, C, C++, Java, Python. This tool is capable of gathering performance information through instrumentation of functions, methods, basic blocks, and statements.
TAU tool uses PAPI for event collection and provides two tools for visualization. The text based tool is called pprof and the graphical tool is called paraprof.
A *very* short guide to using TAU on PACE clusters
* First, you need to recompile your code with TAU wrappers.
- Load the modules your code needs (compiler, MPI, etc)
module load gcc/4.4.5 mvapich2/1.6
- Load the latest tau module (currently tau/2.22-p1, older versions are known to have bugs)
module load tau/2.22-p1
(This will load PDT and PAPI modules too, if you don’t have them loaded already)
- The TAU module will set the correct TAU Makefile in your environment. Check if you have it right:
$ echo $TAU_MAKEFILE
/usr/local/packages/tau/2.22-p1/mvapich2-1.6/gcc-4.4.5/x86_64/lib/Makefile.tau-papi-mpi-pthread-pdt-openmp
• Compile your code using one of the compiler wrapper scripts.
E.g., for a f90 code:
tau_f90.sh -L${PAPIDIR}/lib -lpfm loop_test.f90 -o loop_test
Note that “-L${PAPIDIR}/lib -lpfm” part is necessary on PACE clusters to avoid the system default libpfm, which is not compatible with TAU. If you don’t specify this, you will get this warning:
Error: Reverting to a Regular Make
To suppress this message and revert automatically, please add -optRevert to your TAU_OPTIONS environment variable
Press Enter to continue
* Run the code as usual (not on the headnode!!)
mpirun -np 4 ./loop_test
You will see profiler files in the format “profile.A.B.C” in the same folder, which indicates TAU ran and collected profiling data
* Finally, run pprof or paraprof from the same directory to see the results!
- pprof -ea (sort by exclusive time and show all details)
- paraprof
Remember, these are very brief instructions. Please refer to PAPI and TAU documentation for more details:
Enjoy!