To demonstrate job submission, we will run fastqc on a sample dataset. To do that I need to create a folder in which my job will be running. All analyses are expected to be running on /work file system, check Filesystems replace with your username
mkdir -p /work/$USER/slurm_test/fastqc
cd /work/$USER/slurm_test/fastqc
Check that the needed software is installed,
module av fastqc
# OR
module spider fastqc
In case the software is not available, you can contact the admins to install it. You may have to wait a bit before the software is installed. Otherwise, see if it can be installed via EasyBuild or conda, see Install software. Another possibility is to run it in a container if an image exists. Advanced users can also try to install it locally in "/work/$USER/software". Read the section about installing software.
We assume the software is available, the next step is to write our slurm batch script. We will load the software in that script. With the command below we will make a file called 'slurm_fastqc.sbatch'
cat > slurm_fastqc.sbatch <<EOF
#!/bin/bash
#SBATCH -J test_fastqc # Job name
#SBATCH --cpus-per-task 4 # Total number of threads that can be used in this script
#SBATCH --mem=2000 # Maximum amount of memory in Mb the job can use, 2000 is 2Gb
#SBATCH -t 00:10:00 # Run time ([d-]hh:mm:ss) - 10mins
#SBATCH -o /work/$USER/slurm_test/fastqc/%x.%j.out # Name of stdout output file (%j expands to jobId, %x encodes for the job name)
#SBATCH -e /work/$USER/slurm_test/fastqc/%x.%j.err # Name of stderr output file (%j expands to jobId, %x encodes for the job name)
# Load fastqc module
module load fastqc/0.12.1
# Defining variable for output directory
baseFolder="/work/$USER/slurm_test/fastqc/"
readFolder='/tools/test_data/reads' # Folders with all input files
tmpDir=\${baseFolder}/tmp # It may be interesting to control where the temporary data will be for testing or debugging purposes
outputDir=\${baseFolder}/output # Output directory is where the final data will be after the sofwtare runs
# Creating temporary folder and output folder
mkdir -p \${tmpDir}
mkdir -p \${outputDir}
# Makes sure that input files exist, end the script if it does not
if ! ls \${readFolder}/*.fastq.gz 1> /dev/null 2>&1; then
echo "No .fastq.gz files found in ${readFolder}"
exit 1
fi
# Command to run FastQC
fastqc --dir \${tmpDir} --threads 4 --outdir \${outputDir} \${readFolder}/*.fastq.gz
EOF
To submit the above script to slurm use the command below,
sbatch slurm_fastqc.sbatch
Use the command below to check if your command is in the slurm queue
squeue -u $USER