Software Environment

Default versions have changed for both Cray and 3rd party software, and some software versions are no longer available. Please check the availability and default versions of applications or libraries. You can also check available software with "module avail" on Kraken.

Array jobs on Kraken are no longer supported. The submission filter will reject jobs which make use of job arrays (i.e. #PBS -t or qsub -t). These jobs (if submitted) will not run and should be deleted.

The shell initiation line in PBS scripts is not guaranteed to be used to determine the interpreting shell. The default behavior is to use the user's default login shell or the value of the PBS option -S (i.e. #PBS -S /bin/bash or qsub -S /bin/bash). If you are using a shell for a PBS script which is different than your default shell, please use the PBS -S option.

PVM is a communication interface for parallel programming. While it has been ported to a number of platforms, including some Cray platforms in the past, it has not been ported to the Cray XTs. Thus, we will not install or support PVM on Kraken or Athena. We do allow MPI and Global Arrays on Kraken, as well as pthreads within a single node (this includes OpenMP). It would also be possible to support Unified Parallel C or CoArray Fortran given sufficient demand.

Some sites recommend using the .modulerc file to set your default modules. Do not do so on Kraken

For information on modules, see the modules page.

Different operating systems use different methods of indicating the end of a line in a text file. UNIX uses only a new line, whereas Windows uses a carriage return and a line feed. If you have transferred text files from your PC to a UNIX machine, you may need to remove the carriage-return characters. (The line-feed character in Windows is the same as the new-line character under UNIX, so it doesn’t need to be changed.) Some systems provide a command dos2unix that can perform the translation. However, it can also be done with a simple perl command. In the following example, win.txt is the file transferred from your PC, and unix.txt is the new file in UNIX text format:

perl -p -e 's/\r$//' win.txt unix.txt

If vi appears to hang, but other commands (ls, cat, etc) work normally, try renaming the the .viminfo file:

mv ~/.viminfo ~/.viminfo.bak

This file saves the state of vim, but can sometimes appear to get corrupted due to incompatibilities between different versions of vim

Users may change their default shell in the NICS User Portal . To log into the portal, you need to use your RSA SecurID.

If you haven't already, please check out the other Kraken resource pages at Kraken resources on compiling, file systems, batch jobs, open issues, parallel I/O tips, CrayPAT overview, and other reports and presentations related to Kraken.

Another good resource (without Kraken-specific information) is the documentation that Cray provides at CrayDocs.

Use the following process:

1. Use module load xt-craypat.
2. Compile code.
   • If Fortran90 with modules, compile with -Mprof=func.

At least three profiling tools are available on Kraken.

1. CrayPat is provided by Cray. Follow this link for more information.
2. fpmpi is an unsupported product that can provide a very concise profile of MPI routines in an application. To use it, simply load the fpmpi (or fpmpi_papi) module and relink. Then rerun your application. There are a few environment variables to control profiling output:
   • MPI_PROFILE_DISABLE : Disables statistic collection until fpmpi_enable is called (#include fpmpi.h).
   • MPI_PROFILE_SUMMARY : Setting disables creation of individual MPI process statistics files. Should set this when running with 1000s of processes.
   • MPI_PROFILE_FILE : Name of process statistic file; default is profile.txt.
   • MPI_HWPC_COUNTERS : List of events or event set number as in libhwpc.
3. A third tool that is unsupported is TAU. TAU (Tuning and Analysis Utilities) is a portable profiling and tracing toolkit for performance analysis of parallel programs written in Fortran, C, C++, Java, Python. Basic profiling with TAU can be done in the following steps:
   1. Load the tau module: module load tau
   2. Set the environmental variable TAU_MAKEFILE: In tcsh, setenv TAU_MAKEFILE $(TAUROOT)/lib/Makefile.tau-mpi-pdt
   3. Compile code with the tau wrappers (which should be in your path), tau_f90.sh, tau_cc.sh, or tau_cxx.sh.
   4. You will get a regular executable. Submit your job as usual.
   5. After execution, there should be a profile.xxx text file.

TAU can also do MPI profiling and collect hardware performance counter data.