Jobs with Slurm

The Roar computing clusters are shared computational resources. To perform computationally intensive tasks, users must request compute resources and be provided access to those resources. The request/provision process allows the tasks of many users to be scheduled and carried out efficiently to avoid resource contention. Slurm is utilized by Roar as the job scheduler and resource manager. Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for Linux clusters. Slurm is rapidly rising in popularity and many other HPC systems use Slurm as well. Its primary functions are to

• Allocate access to compute resources to users for some duration of time
• Provide a framework for starting, executing, and monitoring work on the set of allocated compute resources
• Arbitrate contention for resources by managing a queue of pending work

Slurm Resource Directives

Resource directives are used to specify compute resources when submitting a job request to the scheduler. They allow you to indicate the run time, amount of memory, and the number of cores your job will use. They are required when launching Interactive Jobs and Batch Jobs through the command line interface.

Interactive Jobs Through the Roar Portal also use resource directives through the drop down menus in the portal interface.

The following table lists some of the most commonly used resource directives.

Details on additional directives and job options can be found in the Slurm sbatch documentation for batch jobs and the Slurm salloc documentation for interactive jobs.

Replacement Symbols

Replacement symbols can be used inside of slurm directives to populate the values with dynamic content for each job, such as the Job ID or the hostname of the node the job is running on.

Example: Both standard output and standard error are directed to the same file by default, and the file name is slurm-%j.out, where the %j is replaced by the job ID. The output and error filenames are customizable, however, using the table of symbols above.

Additional details on Replacement Symbols can be found in the Slurm sbatch documentation for batch jobs and the Slurm salloc documentation for interactive jobs.

Environmental Variables

Slurm makes use of environment variables within the scope of a job, and utilizing these variables can be beneficial in many cases.

Additional details on Environmental Variables used by Slurm can be found in the Slurm sbatch documentation for batch jobs and the Slurm salloc documentation for interactive jobs.

Hardware details

sinfo

The SLURM command sinfo displays information about all Collab nodes. Its output is more easily read with some formatting options,

sinfo --Format=features:30,nodelist:20,cpus:5,memory:10,gres:30

An example output of the sinfo command would look like:

$ sinfo --Format=features:30,nodelist:20,cpus:5,memory:10,gres:30
AVAIL_FEATURES                NODELIST            CPUS MEMORY    GRES
standard,a100,cascadelake     p-gc-[3001-3035,303848   380000    gpu:a100:2(S:0-11,36-47)
standard,a100_3g,mig,cascadelap-gc-3036           48   380000    gpu:a100_3g:4(S:0-11,36-47)

Details on additional sinfo options can be found in the Slurm sinfo documentation.

Node attributes

Roar Collab contains a wide variety of different hardware configurations. To find out specifics about the hardware on different nodes, there are several helpful tools.

• lscpu: displays information about the CPU and its capabilities
• nvidia-smi: displays information about the GPU (if present).

Using these commands within your jobs can provide details on the hardware it is running on.