Monitor jobs

Contents

Monitor jobs

We assume that you have submitted jobs that are currently running. The idea now is to know the status of your jobs.

Email notifications

If you have used the options --mail-user & --mail-type, you should get email notification when your job reaches a specific status. When --mail-type is set to ALL you will be notified for every status of your job. Here are the possible values for --mail-type,

BEGIN: When job is allocated
END: When job is done
FAIL: When job has fail
REQUEUE: When a fail or finished job is put back into the queue
STAGE_OUT: Burst buffer stage out and teardown completed
TIME_LIMIT: When job has reached 100% of time limit
TIME_LIMIT_90: When job has reached 90% of time limit
TIME_LIMIT_80: When job has reached 80% of time limit
TIME_LIMIT_50: When job has reached 50% of time limit
ARRAY_TASKS: Send emails for each array task
ALL: BEGIN, END, FAIL, REQUEUE, STAGE_OUT

Have a look at the help of the command SBATCH to know more about notifications.

Status of submitted jobs

squeue

Check the status of the queue using.

   1 # Print all jobs currently in the queue
   2 squeue
   3 
   4 # Check owned jobs in the queue
   5 squeue -u <<USERNAME>>
   6 
   7 # Check a list of jobs
   8 squeue --jobs <<JOBID or comma separated list of JOBIDs>>
   9 
  10 # You can format the output of squeue. For example to get more details about the running jobs use,
  11 squeue -o "%.7i %.9P %.8j %.8u %.2t %.10M %.6D %C %.10m"
  12 
  13 # For more info about squeue
  14 man squeue

scancel

Cancel a pending or running job or job step. It can also be used to send an arbitrary signal to all processes associated with a running job or job step.

   1 scancel <<JOBID or comma separated list of JOBIDs>>
   2 
   3 # learn more about scancel option
   4 man scancel

sstat

Prints information about the resources utilized by a running job or job step.

   1 sstat -j <<JOBID or comma separated list of JOBIDs>>
   2 
   3 # learn more about sstat
   4 man sstat

sacct

Reports job or job step accounting information about active or completed jobs.

   1 # Print admin info about a job
   2 sacct -j <<JOBID or comma separated list of JOBIDs>>
   3 
   4 
   5 # Print CPU time and memory usage of a job
   6 sacct --format="CPUTime,MaxRSS" -j <<JOBID>>
   7 
   8 # Print a list of all the job a user has been running
   9 sacct -u <<USERNAME>>
  10 
  11 # learn more about sacct options
  12 man sacct

Status of nodes

Use the command sinfo to report the state of partitions and nodes.

Quick links

#top