• National Institute for Computational Sciences is a UT/ORNL Partnership
Home

Athena

Athena Cray XT4 system specifications:

  • 4,512 compute nodes; four processor cores per node
  • 18,048 compute processors; AMD 2.3 GHz quad-core
  • Each compute node has 4 GB of memory

  • There is a 100 TB raw parallel file system of disk storage for scratch space (85 TB available)
  • Wall clock limit for all queues on Athena is 72 hours
  • The Maximum job size for Athena is 8192 cores


Athena is available for allocation

NSF will continue to fund Athena for an additional year. The number of users given time on Athena will be limited therefore queue wait times should be reduced. Users who would like to try running on Athena are being offered a 50% discount on hours transferred from Kraken until September 30. Athena access includes GSI authentication allowing use of GridFTP. Those interested in transferring hours from Kraken should contact NICS accounts at accounts@nics.tennessee.edu. Athena is currently allocated by direct request to NICS, and allocation requests for projects whose characteristics best match the resource will be selected. Hours may be transferred back to Kraken on request.

For XT system technical details, see Machine size.
This table shows the technical similarities and differences between Athena (XT4 System) and Kraken (Final XT5 System).

Athena and Kraken operate very similarly. Most user guide instructions written for Kraken apply also for Athena. The major difference is that job sizes have to be mulitples of 4 on Athena (12 for Kraken). Note also that Athena has 4GB of memory per node. Additional Differences are indicated below.

User Guides


Athena differences



Connecting

If you have mailed in the notarized form and received email confirmation that your RSA One Time Password (OTP) token has been activated, use ssh to connect to athena.nics.tennessee.edu.

For GSI access to Athena use athena-gsi.nics.utk.edu. For more general information on GSI access please see GSISSH
or Using globus tools. It is also possible to log in using the TeraGrid portal and then use globus tools to gsissh to Athena.



File Systems and Storage

Athena has its own scratch file system (also mounted as /lustre/scratch ). Athena's scratch file system is entirely separate from Kraken's scratch file system. Neither resource can access the other resource's file system. Athena's scratch file system offers 100 TB raw (85 TB available) compared to Kraken which has 3.3 PB raw (2.4 PB Available). As a result, it is imperative that all users archive actively and clean up old data.

Home directories have a 2 GB quota. These directories are on a Network File System (NFS), and are generally available by logging in to login.nics.utk.edu even if Athena is down.

Files from Athena may be stored on the mass storage system (HPSS). You may only access HPSS if you have activated your RSA OTP token. You may access HPSS within a batch job, but only from jobs in the "hpss" queue that are submitted from the RSA SecurID login nodes (athena.nics.tennessee.edu).


Data Transfer

  • When using GSI-SSH authentication, you can move large amounts of data (>1 Gbyte) from another TeraGrid system to Athena using GridFTP: A protocol for very fast file transfers. You can use globus-url-copy (which is similar to scp) or uberftp (which is similar to sftp) to transfer files using GridFTP.
    You may also use globus-url-copy to transfer files from Athena using the file:// syntax: 
     % globus-url-copy -r \
    file:///lustre/scratch/joe/ \
    gsiftp://gridftp-co.ncsa.teragrid.org:2811/joe/
    Note the three slashes.

Software

While the NICS-installed software is generally shared between Kraken and Athena, software from other sources may be slightly different.


Compiling

By default, Athena uses the Portland Group (PGI) compilers. GNU and PathScale compilers are available via modules. To compile your MPI program, use the cc (C), CC (C++), or ftn (Fortran) commands. These commands create executables to run on the compute nodes. Other MPI compiler wrappers such as mpicc, mpicxx, mpif77 and mpi90 are not available on Athena. The Cray scientific libraries (BLAS, LAPACK, ScaLAPACK, BLACS, and others) are in the default library path. Use the module command for access to libraries such as HDF, netCDF, FFTW, and PETSc.

The login nodes are not intended for computing, but if you need to run short pre- or post-processing programs on them, you can call the compilers directly: for PGI compilers, use pgcc, pgCC, or pgf90.


Running Jobs

Athena uses PBS/Torque (qsub, qdel, qstat, etc) for batch jobs. To specify the number cores allocated to a batch job, use "#PBS -l size=cores" in the batch script. Since the batch system allocates in units of cores per node, the number of cores requested must be a multiple of 4.