Submitting jobs

Whatever you read here may need to be adjusted to fit your specific case.

Do not hesitate to ask for some help when needed.

Filesystems

Some but not all partitions are available to the compute nodes. Compute nodes will not be able to access any data from filesystems that are not listed here. /work /scratch /home/

Slurm partitions

There are currently 2 partitions, normal and bigmem.

The normal partition is the default partition if you submit a job without precising which partition should be used. Your job will be placed in the normal partition (250GB).

The normal partition has limited RAM of 250GB, in case you need more than that please use the bigmem partition.

Use -p to specify needed partition.

Load necessary software

By default only some software will be available at login. To be able to use other software scripts you should first load them.

The command module will help you to manage modules and their dependencies.

To check which programs are loaded (ready to be used), use the command below.

module list

Expected output is,

Currently Loaded Modules:
  1) autotools   2) prun/1.3   3) gnu8/8.3.0   4) openmpi3/3.1.4   5) ohpc

To check which programs are available (can be loaded). The same command can be used to search a specific package.

module avail

Expected output is,

--------------------------------------------------- /opt/ohpc/pub/moduledeps/gnu8-openmpi3 ----------------------------------------------------
   adios/1.13.1     hypre/2.18.1    netcdf-cxx/4.3.1        petsc/3.12.0        py2-scipy/1.2.1     scorep/6.0            trilinos/12.14.1
   boost/1.71.0     imb/2018.1      netcdf-fortran/4.5.2    phdf5/1.10.5        py3-mpi4py/3.0.1    sionlib/1.7.4
   dimemas/5.4.1    mfem/4.0        netcdf/4.7.1            pnetcdf/1.12.0      py3-scipy/1.2.1     slepc/3.12.0
   extrae/3.7.0     mpiP/3.4.1      omb/5.6.2               ptscotch/6.0.6      scalapack/2.0.2     superlu_dist/6.1.1
   fftw/3.3.8       mumps/5.2.1     opencoarrays/2.8.0      py2-mpi4py/3.0.2    scalasca/2.5        tau/2.28

-------------------------------------------------------- /opt/ohpc/pub/moduledeps/gnu8 --------------------------------------------------------
   hdf5/1.10.5     metis/5.1.0    mvapich2/2.3.2    openblas/0.3.7        pdtoolkit/3.25      py3-numpy/1.15.3
   likwid/4.3.4    mpich/3.3.1    ocr/1.0.1         openmpi3/3.1.4 (L)    py2-numpy/1.15.3    superlu/5.2.1

------------------------------------------------------------- /tools/modulefiles --------------------------------------------------------------
   MEGAHIT/1.2.9

---------------------------------------------------------- /opt/ohpc/pub/modulefiles ----------------------------------------------------------
   EasyBuild/3.9.4          clustershell/1.8.2    gnu7/7.3.0         llvm5/5.0.1        pmix/2.2.2               valgrind/3.15.0
   autotools         (L)    cmake/3.15.4          gnu8/8.3.0  (L)    ohpc        (L)    prun/1.3          (L)
   charliecloud/0.11        gnu/5.4.0             hwloc/2.1.0        papi/5.7.0         singularity/3.4.1

  Where:
   L:  Module is loaded

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

To search for a module,

module avail <<keyword>>
#OR
module spider <<keyword>>

To load a module do,

module load <<MODULENAME/VERSION>>

Loading a module can be done following these 3 steps,

1. Locate the module, module avail

2. Check how to load it, module spider <<MODULENAME/VERSION>>

3. Load your module using the instructions module load <<MODULENAME/VERSION>>

4. Check that the module is loaded, module list

5. Enjoy!!!

Read more about module usage https://lmod.readthedocs.io/en/latest/010_user.html

Prototype of batch script

This prototype should be in a script file, for example, my_first_script.sbatch

   1 #!/bin/bash
   2 
   3 #SBATCH -J test               # Job name
   4 #SBATCH -o /work/<<UID>>/job.%j.out   # Name of stdout output file (%j expands to jobId)
   5 #SBATCH -e /work/<<UID>>/job.%j.err   # Name of stderr output file (%j expands to jobId)
   6 #SBATCH -n 16                 # Total number of threads requested or total number of mpi tasks
   7 #SBATCH --mem=2000            # Maximum amount of memory in Mb the job can use
   8 #SBATCH -t 00:30:00           # Run time ([d-]hh:mm:ss) - 1.5 hours
   9 #SBATCH --mail-type=ALL
  10 #SBATCH --mail-user=your.email@wur.nl
  11 
  12 # Load your software/command
  13 module load CMD/version
  14 
  15 # Run your command
  16 CMD [OPTIONS] ARGUMENTS

To run a sbatch script use

sbatch <<script name>>

Here are some explanation for obscure elements,

Line starting with #SBATCH

Those lines are option given to sbatch. They are different from the command you are running.

We use %j in the -o option

This is a place holder, it will be replaced by the job id of your run. The use of that makes it easier to find out which standard output correspond to which task. It could be removed but make sure that all the tasks have a specific output file.

--mem num

Omitting this information will result in an exclusive use of a node (1 job uses an entire node). Use this option to set the maximum amount of memory the job can use. Different units can be specified using the suffix [K|M|G|T]. For example, 20G is 20 Gigabits. On normal partition you can use maximum 254000Mb on bigmem you can use 1030055

-t option

The format for the time is, days-hours:minutes:secondes. This sets a limit of time for your task to run. If 00:05:00, your job will run for 5 minutes. What if it is not finished? You will have to rerun it again giving a higher time. If the command you are running has the ability to continue from a checkpoint, you can use that ability to reduce the running time. This parameter is difficult to estimate in most cases, do not hesitate to over estimate at the beginning. A job can run for a maximum of 10 days, 9-23:59:59.

Example of sbtach script

Let's assume a few things here,

1. You need to be logged in
2. Your data is available
3. The needed software is available
4. The test will be run in folder /work/test/

Preparing for the run,

   1 mkdir /work/test/

Let's try to run an assembly using megahit,

   1 #!/bin/bash
   2 
   3 #SBATCH -J test               # Job name
   4 #SBATCH -o /work/test/job.%j.out   # Name of stdout output file (%j expands to jobId)
   5 #SBATCH -e /work/test/job.%j.err   # Name of stderr output file (%j expands to jobId)
   6 #SBATCH -n 16                 # Total number of threads or total number of mpi tasks
   7 #SBATCH --mem=2000            # Maximum amount of memory in Mb the job can use, 2000 is 2Gb
   8 #SBATCH -t 01:30:00           # Run time ([d-]hh:mm:ss) - 1.5 hours
   9 
  10 # Load your available meghit
  11 module load MEGAHIT/1.2.9
  12 
  13 # Defining variable for work directory
  14 based=/work/test
  15 # Defining variable for temporary directory
  16 # We do this because our command uses tmp folder
  17 tmp_dir=$based/tmp
  18 # Defining variable for output directory
  19 output_dir=$based/output
  20 # Defining variable for forward and reverse read file
  21 f_read=/tools/test_data/assembly/r3_1.fa.gz
  22 r_read=/tools/test_data/assembly/r3_2.fa.gz
  23 
  24 # Creating temporary folder
  25 # Megahit will complain if "output" folder already exists
  26 mkdir $tmp_dir
  27 
  28 # Command to run
  29 # We use previously defined varibales to set the values of megahit options
  30 megahit -1 $f_read -2 $r_read --tmp-dir $tmp_dir --out-dir $output_dir --out-prefix r3

Quick links

1. Computers description

2. Request an account

3. storage description

4. First login

5. Receiving sequencing data

6. transfering data

7. Submitting jobs

8. Monitor jobs

9. Install tools