Accessing Compute Resources

The Lovelace Cluster uses Slurm to schedule jobs and allocate resources.

Writing a submission script is typically the most convenient way to submit your job to the job submission system. Example submission scripts (with explanations) for the most common job types are provided below.

Interactive jobs are also available and can be particularly useful for developing and debugging applications. More details are available below.

If you have any questions on how to run jobs on Lovelace do not hesitate to contact the HPC Support at hpcsupport@plymouth.ac.uk.

Using Slurm

You typically interact with Slurm by (1) specifying Slurm directives in job submission scripts (see examples below) and (2) issuing Slurm commands from the login nodes.

There are three key commands used to interact with the Slurm on the command line:

sbatch
squeue
scancel

Check the Slurm man page for more advanced commands:

man slurm

The sbatch command

The sbatch command submits a job to Slurm:

sbatch job_script

This will submit your job script job_script to the job-queues. See the sections below for details on how to write job scripts.

The qstat command

Use the command qstat to view the job queue. For example:

squeue

will list all jobs on Lovelace.

You can view just your jobs by using:

squeue -u <username>

The scancel command

Use this command to delete a job from Lovelace’s job queue. For example:

scancel <jobid>

will remove the job with ID <jobid> from the queue.

Queues

Please note that Slurm job scheduler uses the term ‘partitions’ to refer to queues, and therefore you may see the word partition used interchangeably both here and on other sites.

There are eight partitions available on the Lovelace cluster. These correspond to two sets of four partiitons. These are the standard queues (cpu, cpu_highmem, gpu_h100, gpu_l40s) and the billed queues (cpu-billed, cpu_highmem-billed, gpu_h100-billed, gpu_l40s-billed).

You may only schedule task on the billed queues if there is funding assosciated with your account. Jobs submitted to these queues will be prioritised over those submitted to the standard queue.

`cpu`

To run on the cpu queue, add the following to the header of your submission script :

#SBATCH -p cpu

`cpu_highmem`

To run on the cpu_highmem queue add the following to the header of your submission script :

#SBATCH -p sbatch

`gpu_h100`

To run on the gpu_h100 queue add the following to the header of your submission script :

#SBATCH -p gpu_h100
#SBATCH --gpus 1

Each job must request exactly one GPU if running on this partition.

`gpu_l40s`

To run on the gpu_l40s queue add the following to the header of your submission script :

#SBATCH -p gpu_l40s
#SBATCH --gpus 1

Each job must request exactly one GPU if running on this partition.

Sometimes some nodes are “down” and less nodes are available.

If you have special request, contact hpcsupport@plymouth.ac.uk.

Accounts

Users are allocated to Slurm Accounts based on the projects they are part of. Users that are only part of a single project (and thus a single Slurm Account) will have jobs automatically allocated against this project for accounting purposes. Users that are part of multiple projects should include a line in their submission scripts that sets the account for the job as follows.

#SBATCH --account=<account name>

Output from Slurm jobs

Slurm produces the output of your job submission in a file with the format slurm-<Job ID>.out. This file will be created in the working directory from which you submitted the job with sbatch. You can view the job using the less utility as below:

less -r slurm-<Job ID>.out

Examples of Job Submission Scripts

Some examples are given below:

Run a Python Script using Numpy on a CPU Node

The job submission below assumes that the script is called myscript.py and stored in your home folder.

#!/bin/bash
#SBATCH -p cpu

cd
module load py-numpy
python3 myscript.py

If the submission script is called py-numpy.sbatch, you can submit it by running:

sbatch py-numpy.sbatch

Run a GPU Application on a GPU Node

This job runs the nvidia-smi utility on a GPU node with a GPU, returning details about the allocated GPU

#!/bin/bash
#SBATCH -p gpu_l40s
#SBATCH --gpus 1

nvidia-smi