When you log into Kraken or Athena, you are placed on one of the several login nodes. Login nodes should be used for basic tasks such as file editing, code compilation, data backup, and job submission.
The login nodes should not be used to run production jobs. Production work should be performed on the systems compute resources. Serial jobs (pre- and post-processing, etc.) may be run on the compute nodes as long as they are statically linked. Access to compute resources is managed by the Portable Batch System (PBS). Job scheduling is handled by Moab, which interacts with PBS and the XT system software.
The following pages provide information for getting started with the batch facilities of PBS with Moab as well as basic job execution.
Notice: Compute nodes can see only the Lustre scratch directories.
Batch scripts are run on service nodes that have access to the home, project and software directories. Executables launched with the aprun command do not have access to these directories; they have access only to the Lustre scratch directories. In your batch script, make sure to cd to the Lustre scratch directory before the aprun command is issued. If this is not done, you may see an error like:
aprun: [NID 94]Exec /lustre/scratch/userid/a.out failed: chdir /nics/b/home/userid No such file or directory
For the program launched by aprun, all input and output files must reside in the Lustre scratch directories.Batch Scripts
Batch scripts can be used to run a set of commands on a systems compute partition. The batch scripts page gives example batch scripts, how to submit the batch scripts, and common usage tips.
Interactive Batch Jobs
Interactive batch jobs give users interactive access to compute resources. A common use for interactive batch jobs is debugging. The interactive batch jobs page shows how to run interactive jobs through the batch system and provides common usage tips.
Common PBS Options
The common PBS options page gives a quick overview of common PBS options.
PBS Environment Variables
The PBS environment variables page gives a quick overview of useful environment variable sets within PBS jobs.
Altering Batch Jobs
The altering batch jobs page lists ways to remove or alter batch jobs.
Monitoring Job Status
The monitoring job status page lists ways to monitor jobs in the batch queue.
Queues are used by the batch scheduler to aid in the organization of jobs. The queues page lists the available queues.
Job Execution
Once access to compute resources has been allocated through the batch system, users have the ability to execute jobs on the allocated resources. The job execution page gives examples of job execution and provides common tips.
Job Accounting
Projects are charged based on usage of compute resources. The job accounting page gives details on how we calculate each job’s usage.
Scheduling Policy
The scheduling policy page gives details of the scheduling policies.
