// Running Jobs on Discover using Slurm 

Submit a job

In general, you will create a batch job script. Either a shell script or a Python script is allowed, but throughout the user guide we use only shell scripts for demonstration. You will then submit the batch script using sbatch; the following example requests 2 nodes with at least 2GBs of memory per CPU: $ sbatch --nodes=2 --mem-per-cpu=2048 [my_script_file]

The "my_script_file" should include all the necessary requirements for a batch script. Look at the example job scripts here for more information.

At the bottom is a subset of sbatch environment variables which may be useful to NCCS users.

See How to Determine Memory Usage in Using Discover for various methods of calculating your job memory requirements.

Note: The "sbatch" command scans the lines of the script file for SBATCH directives. A line in the script file will be processed as a directive to "sbatch" if and only if the string of characters starts with #SBATCH (with no preceding blanks). The remainder of the directive line consists of the options to "sbatch" in the same syntax as they appear on the command line.

The sbatch command reads down the shell script until it finds the first line that is not a valid SBATCH directive, then stops. The rest of the script is the list of commands or tasks that the user wishes to run.

There are many options to the "sbatch" command. The table lists a few commonly used options. Please refer to the man pages on Discover for additional details.

SBATCH OPTIONS

Submit an interactive job

Use the salloc command to request interactive Discover resources through Slurm. The following command gives you a 3-node job allocation, and places you in a shell session on its head node. Your terminal bell will ring to notify you when you receive your job allocation:

$ salloc --nodes=3 --bell

The options described in the sbatch link below also apply to salloc.

Here is a subset of salloc environment variables which may be most useful to NCCS users.

SBATCH OPTIONS

Submit an interactive job with X 11 forwarding

The following xalloc command (an NCCS wrapper for salloc) sets up X11 forwarding and starts a shell on the job's head node, while the --ntasks argument lets Slurm allocate any number of nodes to the job that together can provide 56 cores:

$ xalloc --ntasks=56

The xalloc wrapper forwards all options to salloc. Please note: both salloc and xalloc will place you on the head node of a multi-node allocation.

Submit a set of jobs

Using: --depend

Quite often, users may want to execute multiple long runs which must be processed in sequence. SBATCH job dependencies allow you to prevent a job from running until another job has completed one of several actions (started running, completed, etc...).

SBATCH allows users to move the logic for job chaining from the script into the scheduler. The format of a SBATCH dependency directive is -d, --dependency=dependency_list , where dependency_list is of the form: type:job_id[:job_id][,type:job_id[:job_id]] For example,

$ sbatch --dependency=afterok:523568 secondjob.sh

only schedules the second job after the Job 523568 was successfully completed. Useful "types" in the dependency expression are

• afterok: Job is scheduled if the Job exits without errors or is successfully completed.
• afternotok: Job is scheduled if the Job exits with errors.
• afterany: Job is scheduled if the Job exits with or without errors.
• after: Job is scheduled if the Job has started.

A simple example below shows how to submit three batch jobs with dependencies, using the following driver script:

#!/bin/bash
FIRST=$(sbatch testrun.sh | cut -f 4 -d' ')
echo $FIRST
SECOND=$(sbatch -d afterany:$FIRST testrun.sh | cut -f 4 -d' ')
echo $SECOND
THIRD=$(sbatch -d afterany:$SECOND testrun.sh | cut -f 4 -d' ')
echo $THIRD
exit 0

The SECOND job will be on hold until the FIRST job exits with or without error. The THIRD job will be on hold until the SECOND job completes.

The second example below shows a script to submit a job "job2.slurm" but the job will not be queued until the current running or queued jobs are all completed with no errors.

#!/usr/bin/csh -fx
# Query all my jobs (squeue -u) and reformat the job ids into
# a string with the form: Job-ID1: Job-ID2: Job-ID3…
set arglist = `squeue -u myuserid --noheader -o "%i" | \
sed -n '1h;2,$H;${g;s/\n/:/g;p}'`
sbatch -d after:$arglist job2.slurm
exit 0

Using: --nice

The sbatch "nice" option can be assigned a value of 1 to 10000, where 10000 is the lowest available priority. (This value specifies a scheduling preference among a set of jobs, but it is still possible for Slurm's backfill algorithm to run a lower-priority job before a higher priority job. For strict job ordering, use --depend as described above.) To specify a low-priority job in relationship to a higher-priority, important job:

$ sbatch ./important.sh
$ sbatch --nice=10000 ./low_priority.sh

Submit replicated jobs

A job array represents a collection of subjobs (also referred to as job array "tasks") which only differ by a single index parameter. Sometimes users may want to submit many similar jobs based on the same job script. Rather than using a script or multiple similar scripts and repeatedly calling sbatch, a job array allows the creation of multiple such subjobs within one job script. Therefore, it offers users a mechanism for grouping related work, making it possible to not only submit, but also query, modify and display the set as a single unit. (See additional job-array monitoring and control methods in man pages for squeue, scontrol, and scancel.)

Job arrays are submitted through the -a or --array option to sbatch. This option specifies multiple values using a comma separated list and/or a range of values with a "-" separator. An example job array script "testArrayjobs.sh" is shown below.

#!/bin/bash
#SBATCH -N Test_Arrayjobs
#SBATCH --ntasks-per-node=12 --ntasks=24
#SBATCH --constraint=hasw
#SBATCH --time=12:00:00
#SBATCH -o output.%A_%a
#SBATCH --account=xxxx
#SBATCH --array=0-4:2
. /usr/share/modules/init/bash
module purge
module load comp/intel-12.1.0.233 mpi/impi-4.1.0.024
cd /discover/nobackup/myuserid
mkdir -p work.${SLURM_ARRAY_TASK_ID}
cd work.${SLURM_ARRAY_TASK_ID}
cp /discover/nobackup/myuserid/myexec .
echo ${SLURM_ARRAY_TASK_ID} > input.${SLURM_ARRAY_TASK_ID}
mpirun -np 24 ./myexec exit 0


In this case, "sbatch testArrayjobs.sh" submits a job array including three subjobs, each with a unique sequence number 0, 2, and 4 defined by "-a" or "--array=0-4:2". Each of the 3 subjobs are running in different work directories: work.0, work.2, or work.4. Each subjob runs on 2 nodes and 24 MPI tasks, reading in different input files and writing standard output to output.jobid_array_task_index. Note that "%A" is replaced by the job ID and "%a" with the array task ID.

$ squeue | grep myuserid
524230_0 compute Test_Arra myuserid R 0:03 2 borgr[019-020]
524230_2 compute Test_Arra myuserid R 0:03 2 borgr[001-002]
524230_4 compute Test_Arra myuserid R 0:03 2 borgr[003-004]

Environment variables for sbatch, salloc, and srun

Slurm provides extensive automation and customization capabilities for each of its commands, through a set of environment variables. We link to the essential subsets of environment variables for sbatch, salloc in the examples below, and recommend these links as starting points. If these are inadequate for your needs, then for a complete, detailed list of environment variables, see the man page for each command, or contact NCCS support.

SBATCH

SALLOC

SRUN

Learn how to use bash and c-shell scripts to automate jobs, using example scripts as a reference.