No sub-articles
Browse articles in this level »
New article next to 4 Install software New article below 4 Install software
Table-of-Content

Introduction

!WARNING!: This page of the tutorial is meant for experienced users. Beginners to Linux can request software install at bioinfo.mib@wur.nl and students may get help from their supervisors.

A cluster environment typically restrict users from installing software system-wide due to security and stability concerns. However, most HPC systems allow users to compile and install software in their own directories. This guide explains how to install software locally in the /work/$USER/software directory, a common practice for custom software installations on HPC clusters.

Every software has it own specificities when it comes to installation, you need to refer to the software documentation to know the requirements and the steps to install it.

Why Install Software Locally?

Users often need specific software versions or packages that are not yet available through the general module system.

Local installation provides:

Version control: Install specific versions required for your research

Customization: Configure software with specific options and dependencies, the local installation sometimes gives more flexibility

Independence: Avoid waiting for system administrators to install packages

Experimentation: Test new software without affecting other users

Containers

Some software offer a containerized environment, which means that the sofwtare and all it's dependencies (except maybe database) are already installed in a package that users can directly use. To run containers load the module apptainer,

module load apptainer

In some rare cases, you may need to convert the image before using it. The software website usually describes how to proceed. Otherwise, take contact with bioinfo.mib@wur.nl

examples of software that you can run using containers can be found in the analysis guide (e.a Qiime, GTDB-Tk)

Prerequisites and Considerations

  • Avoid installing software in your home folder
  • Avoid installing software that are already installed
  • We ask you to always install your local software in the folder, /work/$USER/software/ this folder is only writable by you and visible by everybody else. You can make with the commands below,
mkdir /work/$USER/software/                          # make the folder
chmod go-w,go+rx /work/$USER/software/   # Adjust the permissions if needed

  • Databases should be installed separately in /work/databases/, make sure the one you need is not already installed.

  • When binaries are available, choose the one for redhat, rhel, rocky Linux or Linux

  • It is always better to compile software on a compute node, to make sure all the requirements are present before submitting jobs.

  • Keep your software folder organised, we recommend creating this hierarchy Class/NameOfSoftware/VersionNumber for all software The 'Class' can refer to different categories related to the software. It could be as generic as 'bioinformatics' or 'system' or it could be specific like 'QC', 'assembly', 'phylogenetics'

  • Load the appropriate modules before compiling or installing software otherwise the system libraries will be used and those could be different between nodes. 

module load gcc/14.3.0
module load cmake/4.3.2
module load java
module load python/3.14.4
module load mpi/openmpi-x86_64
...

Installing from source

Some software only distribute the sources meaning the code. Before using it you need to compile it, that usually takes 2 steps configure and build. It is critical to set the prefix to the local folder in the configure step otherwise the system will try to install it in one of the system folder where users often don't have access. Besides the prefix there could be other important modules to configure, it is up to the person installing to find and set those up.

Configure the Build

Use the '--prefix' option to direct installation to your user directory

For GNU Build System (configure/make):

./configure --prefix=/work/$USER/software/Class/NameOfSoftware/VersionNumber

For CMake-based projects:

mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX:PATH=/work/$USER/software/Class/NameOfSoftware/VersionNumber ..

For Python packages: (Not recommended)

pip install --prefix=/work/$USER/software/Class/NameOfSoftware/VersionNumber

After the configure step, usually comes the build and install step. There could be an optional test step in between. 

make             # this command builds the software
make test      # Test the software before installing, this step is optional
make install # Install the software in the PREFIX folder

Use your newly installed software

Once the software has been built and installed, new folder appears in the installation folder. A 'bin' folder will almost always be created, optionally 'lib', 'lib64', 'include', 'share' folders may be created. At this point you can use your software by using the absolute path to the binary /work/$USER/software/Class/NameOfSoftware/VersionNumber/bin/binaryName.

To avoid typing the entire path of a binary you can set it up in the ~/.bashrc, choose the commands that suit your case.Alternatively you can also create a module file and load it like other modules on the system (module avail, module load).

With ~/.bashrc

Be careful with your bashrc as some errors could have you locked out of the system.

Add to PATH for executables:

echo "export PATH=/work/$USER/software/Class/NameOfSoftware/VersionNumber/bin:$PATH" >> ~/.bashrc

For libraries, set LD_LIBRARY_PATH:

echo "export LD_LIBRARY_PATH=/work/$USER/software/Class/NameOfSoftware/VersionNumber/lib:$LD_LIBRARY_PATH" >> ~/.bashrc

For header files (development):

echo "export CPATH=/work/$USER/software/Class/NameOfSoftware/VersionNumber/include:$CPATH" >> ~/.bashrc echo "export LIBRARY_PATH=/work/$USER/software/Class/NameOfSoftware/VersionNumber/lib:$LIBRARY_PATH" >> ~/.bashrc

For pkg-config:

echo "export PKG_CONFIG_PATH=/work/$USER/software/Class/NameOfSoftware/VersionNumber/lib/pkgconfig:$PKG_CONFIG_PATH" >> ~/.bashrc

For CMake to find the package:

echo "export CMAKE_PREFIX_PATH=/work/$USER/software/Class/NameOfSoftware/VersionNumber:$CMAKE_PREFIX_PATH" >> ~/.bashrc

With module files

This option maybe easier for software management and will prevent many probable errors in your bashrc file. We need a separate folder for module files, let's use /work/$USER/software/modulefiles/, we keep the same organisation as for the software installation /work/$USER/software/modulefiles/Class/NameOfSoftware/VersionNumber

  • Create the file and adjust the content to have a valid module file, you may remove lines that are not relevant.
cat » /work/$USER/software/modulefiles/Class/NameOfSoftware/VersionNumber <<EOF 
#%Module1.0#####################################################################

proc ModulesHelp { } {
    puts stderr " "
    puts stderr "This module loads NameOfSoftware"
    puts stderr " "
    puts stderr "A one line description of the software"
    puts stderr " "
    puts stderr "\nVersion 0.12.1\n"
}
module-whatis "Name: NameOfSoftware"
module-whatis "Version: 0.12.1"
module-whatis "Description: A more extensive description of the software. All in one line"
module-whatis "URL: https://github.com/link/to/software"

set     version                                                  0.12.1
set     rootd                                                      /work/$USER/software/Class/NameOfSoftware/$version
set     SOFTWARE_NAME_HOME             $rootd
prepend-path    PATH                                   $rootd/bin
prepend_path   LD_LIBRARY_PATH        $rootd/lib
prepend_path   LD_LIBRARY_PATH         $rootd/lib64
prepend_path   CPATH                                 $rootd/include

prepend_path   LIBRARY_PATH                $rootd/lib
prepend_path   PKG_CONFIG_PATH        $rootd/lib/pkgconfig
prepend_path   CMAKE_PREFIX_PATH   $rootd

# Load other dependencies
module load java
module load another-dependency

family "user_Class_NameOfSoftware"

EOF
  • To use this custom module folder, run the command
module use /work/$USER/software/modulefiles/
  • After the above step you, these module can be searched and loaded
module avail
module load NameOfSoftware/Version

Installing python tools/modules

We recommend installing Python software in its own folder. To do this, we create a virtual environment for each piece of software that we want to install, and then use a Python package manager to install it. It is important to use the correct version of Python from the start.

modue load python/3.14.4
python -mvenv --copies /work/$USER/software/Class/NameOfSoftware/VersionNumber/
#OR
uv venv --relocatable /work/$USER/software/Class/NameOfSoftware/VersionNumber/
  • Once a Python virtual environment has been created, it cannot be moved or renamed. After creating the environment, you can simply activate it and use Pip to install your package. Messages about upgrading pip could be ignored or followed.
source /work/$USER/software/Class/NameOfSoftware/VersionNumber/bin/activate
pip install NameOfSoftware
  • The module could be used in the slurm sbatch scripts after activating the environment like above, but we recommend creating a module file and load it as any other module
cat » /work/$USER/software/modulefiles/Class/NameOfSoftware/VersionNumber <<EOF 
#%Module1.0#####################################################################

proc ModulesHelp { } {
    puts stderr " "
    puts stderr "This module loads PythonModule"
    puts stderr " "
    puts stderr "A one line description of the software"
    puts stderr " "
    puts stderr "\nVersion 0.12.1\n"
}

module-whatis "Name: NameOfSoftware"
module-whatis "Version: 0.12.1"
module-whatis "Description: A more extensive description of the software. All in one line"
module-whatis "URL: https://github.com/link/to/software"

set     version                                                  0.12.1
set     rootd                                                      /work/$USER/software/Class/NameOfSoftware/$version
set     SOFTWARE_NAME_HOME             $rootd

# Load other dependencies
module load python/version
module load another-dependency

family "user_Class_NameOfSoftware"

# Source the python virtual environment, only for python virtual environment
# On load, activate the environment for the current python
if { [ module-info mode load ] } {
    if { [ module-info shell ] == "bash" } {
        puts stdout "source $rootd/bin/activate;"
    }
}

# On unload, deactivate any active environment and clean up probable functions
if { [ module-info mode remove ] } {
    puts stdout "deactivate"
}

EOF
  • Load the module as described above

Install via EasyBuild

echo "export XDG_CONFIG_DIRS="/tools/sw/compilers/easybuild/5.3.1${XDG_CONFIG_DIRS:+:$XDG_CONFIG_DIRS}" » /etc/profile.d/sh.local

Install tools with conda