Submitting jobs

Contents

Submitting jobs

Whatever you read here may need to be adjusted to fit your specific case.

Do not hesitate to ask for some help when needed.

Filesystems

Some but not all partitions are available to the compute nodes. Compute nodes will not be able to access any data from filesystems that are not listed here.

/work
/scratch
/home

Slurm partitions

There are currently 2 partitions, normal and bigmem.

The normal partition is the default partition if you submit a job without precising which partition should be used. Your job will be placed in the normal partition (250GB).

The normal partition has limited RAM of 250GB, in case you need more than that please use the bigmem partition.

Use -p to specify needed partition.

Load necessary software

By default only some software will be available at login. To be able to use other software scripts you should first load them.

The command module will help you to manage modules and their dependencies.

To check which programs are loaded (ready to be used), use the command below.

module list

Expected output is,

Currently Loaded Modules:
  1) autotools   2) prun/1.3   3) gnu8/8.3.0   4) openmpi3/3.1.4   5) ohpc

To check which programs are available (can be loaded). The same command can be used to search a specific package.

module avail

Expected output is,

--------------------------------------------------- /opt/ohpc/pub/moduledeps/gnu8-openmpi3 ----------------------------------------------------
   adios/1.13.1     hypre/2.18.1    netcdf-cxx/4.3.1        petsc/3.12.0        py2-scipy/1.2.1     scorep/6.0            trilinos/12.14.1
   boost/1.71.0     imb/2018.1      netcdf-fortran/4.5.2    phdf5/1.10.5        py3-mpi4py/3.0.1    sionlib/1.7.4
   dimemas/5.4.1    mfem/4.0        netcdf/4.7.1            pnetcdf/1.12.0      py3-scipy/1.2.1     slepc/3.12.0
   extrae/3.7.0     mpiP/3.4.1      omb/5.6.2               ptscotch/6.0.6      scalapack/2.0.2     superlu_dist/6.1.1
   fftw/3.3.8       mumps/5.2.1     opencoarrays/2.8.0      py2-mpi4py/3.0.2    scalasca/2.5        tau/2.28

-------------------------------------------------------- /opt/ohpc/pub/moduledeps/gnu8 --------------------------------------------------------
   hdf5/1.10.5     metis/5.1.0    mvapich2/2.3.2    openblas/0.3.7        pdtoolkit/3.25      py3-numpy/1.15.3
   likwid/4.3.4    mpich/3.3.1    ocr/1.0.1         openmpi3/3.1.4 (L)    py2-numpy/1.15.3    superlu/5.2.1

------------------------------------------------------------- /tools/modulefiles --------------------------------------------------------------
   MEGAHIT/1.2.9

---------------------------------------------------------- /opt/ohpc/pub/modulefiles ----------------------------------------------------------
   EasyBuild/3.9.4          clustershell/1.8.2    gnu7/7.3.0         llvm5/5.0.1        pmix/2.2.2               valgrind/3.15.0
   autotools         (L)    cmake/3.15.4          gnu8/8.3.0  (L)    ohpc        (L)    prun/1.3          (L)
   charliecloud/0.11        gnu/5.4.0             hwloc/2.1.0        papi/5.7.0         singularity/3.4.1

  Where:
   L:  Module is loaded

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

To search for a module,

module avail <<keyword>>
#OR
module spider <<keyword>>

To load a module do,

module load <<MODULENAME/VERSION>>

Loading a module can be done following these 3 steps,

Locate the module, module avail
Check how to load it, module spider <<MODULENAME/VERSION>>
Load your module using the instructions module load <<MODULENAME/VERSION>>
Check that the module is loaded, module list
Enjoy!!!

Read more about module usage https://lmod.readthedocs.io/en/latest/010_user.html

Prototype of batch script

This prototype should be in a script file, for example, my_first_script.sbatch

   1 #!/bin/bash
   2 
   3 #SBATCH -J test               # Job name
   4 #SBATCH -o /work/<<UID>>/job.%j.out   # Name of stdout output file (%j expands to jobId)
   5 #SBATCH -e /work/<<UID>>/job.%j.err   # Name of stderr output file (%j expands to jobId)
   6 #SBATCH -n 16                 # Total number of threads requested or total number of mpi tasks
   7 #SBATCH --mem=2000            # Maximum amount of memory in Mb the job can use
   8 #SBATCH -t 00:30:00           # Run time ([d-]hh:mm:ss) - 1.5 hours
   9 #SBATCH --mail-type=ALL
  10 #SBATCH --mail-user=your.email@wur.nl
  11 
  12 # Load your software/command
  13 module load CMD/version
  14 
  15 # Run your command
  16 CMD [OPTIONS] ARGUMENTS

To run a sbatch script use

sbatch <<script name>>

Here are some explanation for obscure parts of this template,

Line starting with #SBATCH: Those lines are option given to sbatch. They are different from the command you are running.
We use %j in the -o option: This is a place holder, it will be replaced by the job id of your run. The use of that makes it easier to find out which standard output correspond to which task. It could be removed but make sure that all the tasks have a specific output file.
--mem num: Omitting this information will result in an exclusive use of a node (1 job uses an entire node). Use this option to set the maximum amount of memory the job can use. Different units can be specified using the suffix [K|M|G|T]. For example, 20G is 20 Gigabits. On normal partition you can use maximum 254000Mb on bigmem you can use 1030055
-t option: The format for the time is, days-hours:minutes:secondes. This sets a limit of time for your task to run. If 00:05:00, your job will run for 5 minutes. What if it is not finished? You will have to rerun it again giving a higher time. If the command you are running has the ability to continue from a checkpoint, you can use that ability to reduce the running time. This parameter is difficult to estimate in most cases, do not hesitate to over estimate at the beginning. A job can run for a maximum of 10 days, 9-23:59:59.

Example of sbtach script

Let's assume a few things here,

You need to be logged in
Your data is available
The needed software is available
The test will be run in folder /work/test/

Preparing for the run,

   1 mkdir /work/test/

Let's try to run an assembly using megahit,

   1 #!/bin/bash
   2 
   3 #SBATCH -J test               # Job name
   4 #SBATCH -o /work/test/job.%j.out   # Name of stdout output file (%j expands to jobId)
   5 #SBATCH -e /work/test/job.%j.err   # Name of stderr output file (%j expands to jobId)
   6 #SBATCH -n 16                 # Total number of threads or total number of mpi tasks
   7 #SBATCH --mem=2000            # Maximum amount of memory in Mb the job can use, 2000 is 2Gb
   8 #SBATCH -t 01:30:00           # Run time ([d-]hh:mm:ss) - 1.5 hours
   9 
  10 # Load your available meghit
  11 module load MEGAHIT/1.2.9
  12 
  13 # Defining variable for work directory
  14 based=/work/test
  15 # Defining variable for temporary directory
  16 # We do this because our command uses tmp folder
  17 tmp_dir=$based/tmp
  18 # Defining variable for output directory
  19 output_dir=$based/output
  20 # Defining variable for forward and reverse read file
  21 f_read=/tools/test_data/assembly/r3_1.fa.gz
  22 r_read=/tools/test_data/assembly/r3_2.fa.gz
  23 
  24 # Creating temporary folder
  25 # Megahit will complain if "output" folder already exists
  26 mkdir $tmp_dir
  27 
  28 # Command to run
  29 # We use previously defined varibales to set the values of megahit options
  30 megahit -1 $f_read -2 $r_read --tmp-dir $tmp_dir --out-dir $output_dir --out-prefix r3

YOUR ATTENTION PLEASE! Once your job is submitted, slurm takes care of it until it is done. There is no need to run your job in screen or any special terminal.

Use the node local storage (2TB)

As the title suggests, every compute node comes with a local storage (only visible on the node). Using that space can make your analysis faster but it must be used with care to not loose any data. It is important to understand that this space is created when a job or a task starts to run. The space is specific of the job or task. So for jobs that run over multiple nodes this might not be a solution. Finally, the space is DELETED as soon as your job is finished (no one can access it anymore)

To reach this space use the variable $TMPDIR in your sbatch script.

Here are 2 ways to use the local storage:

Run your enntire analysis in that folder and copy the results back to your work folder

Here is how to do this in practice, In this example we will run a spades command entirely on the compute node then copy the output back to our /work folder. To avoid many troubles all paths should be in absolute forms.

Rather than copying the entire output, it might be wiser to only take the files that are needed for next steps of analysis. It is also a good idea to assess whether the run completed in a good way before copying files back.

   1 #!/bin/bash
   2 
   3 #SBATCH --mail-user=your.email@wur.nl
   4 #SBATCH --mem 3000
   5 #SBATCH -n 16
   6 #SBATCH -t 00:15:00
   7 #SBATCH -D $TMPDIR # slurm automatically moves to this folder before the run starts
   8 #SBATCH -J MYJOB  # The name of my job, could be any name
   9 #SBATCH -o $TMPDIR/MYJOB.out
  10 #SBATCH -e $TMPDIR/MYJOB.err
  11 
  12 # Load spades
  13 module load spades;
  14 
  15 # Make a directory for temporary files
  16 mkdir $TMPDIR/spades_tmp
  17 
  18 # Make a directory for output of spades
  19 mkdir $TMPDIR/spades_out
  20 
  21 # Run spades, please adjust the resource allocation to your needs
  22 spades.py --meta -t 16 -m 3 \
  23 --tmp-dir $TMPDIR/spades_tmp \
  24 -1 /work/username/forward.fastq.gz \
  25 -2 /work/username/reverse.fastq.gz \
  26 -o $TMPDIR/spades_out
  27 
  28 # Copy the results back to user's work folder. Can be done using the command "cp -r" as well
  29 rsync -rt $TMPDIR/spades_out /work/username/
  30 
  31 # Copy the slurm output files back to user's work folder
  32 cp $TMPDIR/MYJOB.out $TMPDIR/MYJOB.err  /work/username/

Setup the program you run to create temporary files on that location

In this case, we assume that the program you intend to run offer the option to specify a location for temporary files. For example, the assembler SPAdes offers the option '--tmp-dir'. For example,

   1 #!/bin/bash
   2 
   3 #SBATCH --mail-user=your.email@wur.nl
   4 #SBATCH --mem 3000
   5 #SBATCH -n 16
   6 #SBATCH -t 00:15:00
   7 #SBATCH -D /work/username/output
   8 #SBATCH -J MYTEST
   9 #SBATCH -o /work/username/output/MYTEST.out
  10 #SBATCH -e /work/username/output/MYTEST.err
  11 
  12 # Load spades
  13 module load spades;
  14 
  15 # Make a directory for temporary files
  16 mkdir $TMPDIR/spades_tmp
  17 
  18 # Run spades, please adjust the resource allocation to your needs
  19 spades.py --meta -t 16 -m 3 \
  20 --tmp-dir $TMPDIR/spades_tmp \
  21 -1 /work/username/forward.fastq.gz \
  22 -2 /work/username/reverse.fastq.gz \
  23 -o /work/username/output/spades_out

Run multiple individual samples

It sometimes happen that we need to run the same command with different input files and maybe different parameters. This is often the case with metagenomic samples for example. In this case, 2 options are possible write a script for all your samples and submit them all or write a script that will make the individual script per sample and submit them. The last one is the one we develop in this section. It is important to be comfortable with bash scripts, to be able to make the few adjustments that are needed to run your own commands.

   1 #!/usr/bin/env bash
   2 
   3 set -e
   4 
   5 fastqs=$1; shift
   6 suffix=$1; shift
   7 output=$1; shift
   8 
   9 if [ -z $fastqs ]; then suffix='./'; fi
  10 if [ -z $suffix ]; then suffix='.fastq.gz'; fi
  11 if [ -z $output ]; then suffix='./'; fi
  12 
  13 
  14 for fl in $(find $fastqs -name "*_[12]$sufix" | sort | paste -d',' - -)
  15 do
  16  fq1=$(echo $fl | cut -d',' -f1)
  17  fq1=$(realpath $fq1)
  18  fq2=$(echo $fl | cut -d',' -f2)
  19  fq2=$(realpath $fq2)
  20  bs=$(basename $fq1 1$suffix)
  21  bs=$(echo $bs | sed -r "s/[,\._\-\+\=\@\s]+$//")
  22 
  23  o=$(realpath $output/$bs)
  24 
  25  if [ ! -d $o ] ; then mkdir $o; fi
  26 
  27  cat >./${bs}.sbatch <<EOF
  28 #!/bin/bash
  29 
  30 #SBATCH --mail-user=your.email@wur.nl
  31 #SBATCH --mem 3000
  32 #SBATCH -n 10
  33 #SBATCH -t 00:15:00
  34 #SBATCH -D $o
  35 #SBATCH -J $bs
  36 #SBATCH -o ${bs}.out
  37 #SBATCH -e ${bs}.err
  38 
  39 module load fastqc;
  40 
  41 fastqc -o ${o} -t 10 -f fastq $fq1 $fq2
  42 EOF
  43 
  44 #sbatch ./${bs}.sbatch
  45 done

This is a simple script that look for specific pair of "fastq" files filtered by their extensions, will sort the file names and write a sbatch script to run the program "fastqc" on them. Let's see in a bit more detailed what the script does

Line 1

Is the shebang line, it tells which program should be used to run the current script.

Line 3

Sets a parameter for bash so that if any error in the lines below will immediately stop the script.

Line 5-7

Will collect users arguments, the program expects the folder where to find the fastq files as first arguments. Then the full extension of the FASTQ files (.fq or .fastq or .fq.gz...) and at last the output directory where all the output of the individual sbatch scripts will be placed.

Line 9-11

In case no arguments is given the script will set some default values. It assumes that the fastq files and the output will be placed in the current folder.

Line 14

This is a complex line but to put it simple, that line collects the fastq pairs seperated by a comma (forward,reverse) on the same line. The fastq files usually follow the same scheme for their names, SampleID_ADDITIONAL-INFO_[R]1.fastq[.gz] & SampleID_ADDITIONAL-INFO_[R]2.fastq[.gz] the elements between square brackets are optional.

On that line we collect all the fastq files in the folder specified by the user based on their extensions (find command) then we sort the list of files we obtained hoping that we end with pair of files belonging to the same sample with the forward on top of the reverse file (sort command). At last, the command paste will merge every 2 lines into 1, separated by a comma.

Be aware that if the suffix is not specific of the fastq files other files might appear in the list preventing the script from working correctly.

Line 16-19

Makes sure that the path for the forward & reverse read files is in absolute form. It is easier to work with absolute path in a script.

Line 20-21

From the forward read path, it isolate the file name and removes the suffix (basename command) then any trailing spacing character (sed command). The result is use to identify the analysis for each sample. If my filename is SampleID_ADDITIONAL-INFO_R1.fastq.gz the result of these lines will be SampleID_ADDITIONAL-INFO_R

Line 23

The output folder for the current sample will be, the output folder specified by the user plus the ID computed above

Line 25

Create a new folder for the current sample

Line 27-42

The sbatch script for each sample is generated. Refer to the information above and the manual of the command sbatch to understand the script. You will need to adjust this part to your own case, the command to run is of course important but also the resources and time needed to run your samples should be set.

Line 44

This line is commented out to avoid running mistakes. Remember this is a script makes a sbatch script for many samples to limit waste of resources please check that 1 of the sbatch script works well before submitting the rest.

Quick links

#top