Skip to content


Aim: Provide information about how to use containers on Nikhef's computing infrastructure to run different operating systems.

Target audience: Users of the Stoomboot cluster.


Containers allow you to run another operating system distribution (i.e., Linux distributions like CentOS, Ubuntu, etc.) inside a server. Containers are very useful especially if 1. your collaboration/experiment provides containers for performing analysis runs (usually made available by CernVMFS). 2. the analysis has specific operating system requirements, i.e., CentOS 6 or another OS distribution.

Nikhef does not allow superuser privileges (aka root privileges) on Nikhef machines (except our laptops). Apptainer, the Linux Foundation variant of Singularity, in user mode is the preferred container solution.


  • A Nikhef account;
  • An ssh client.


Getting started with Apptainer

The Apptainer binary is available on CVMFS from the Open Science Grid: /cvmfs/ It makes sense to add the path to the Apptainer binary to your path to avoid very long commands in your scripts.

export PATH=$PATH:/cvmfs/

Apptainer downloads images to a cache directory, which can quickly fill up your home directory (the default cache directory). To avoid this, set APPTAINER_CACHEDIR to a path on /data

export APPTAINER_CACHEDIR=/data/my_group/my_username/.apptainer_cache

Adding directories to the container

To access directories outside the working directory, Apptainer can be told to bind-mount the external directory using the -B argument. The value is a comma-separated list of outside_directory:inside_directory:

> apptainer run -B /project/my_project/my_username:/project,/dcache:/dcache docker://ubuntu:latest
INFO:    Using cached SIF image
INFO:    Converting SIF file to temporary sandbox...
WARNING: underlay of /etc/localtime required more than 50 (66) bind mounts
Apptainer> ls /dcache
alice  antares  atlas  auger  bfys  datagrid  detrd  etseis  gravwav  hisparc  km3net  test  theorie  xenon

Running the Container

Apptainer supports running docker containers:

> apptainer run docker://ubuntu:latest
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 7b1a6ab2e44d done
Copying config e7132beceb done
Writing manifest to image destination
Storing signatures
2021/12/03 13:36:54  info unpack layer: sha256:7b1a6ab2e44dbac178598dabe7cff59bd67233dba0b27e4fbd1f9d4b3c877a54
INFO:    Creating SIF file...
INFO:    Converting SIF file to temporary sandbox...
WARNING: underlay of /etc/localtime required more than 50 (66) bind mounts
Apptainer> pwd
Apptainer> cat /etc/os-release
VERSION="20.04.3 LTS (Focal Fossa)"
PRETTY_NAME="Ubuntu 20.04.3 LTS"

Repeatedly Running Containers

To speed up the start of Apptainer by avoiding repeated builds of the containers, you can build an image:

> apptainer build --sandbox --fix-perms /data/my_group/my_username/ubuntu_latest docker://ubuntu:latest

And then start it with

> apptainer run /data/my_group/my_username/ubuntu_latest

The --writable argument can be passed to apptainer run, which will allow installation of additional software in the image. Important caveat: the installation of software should not try to change the user (which is not allowed).

Building and installing software in shared directories is possible, but in this case, the path inside and outside of the container should be the same to avoid issues with relocation of built libraries and hard-coded paths.

Adding GPU resources to the container

To get access to the GPUs in the system, pass either --rocm or --nv to apptainer run as needed. For example: Tensorflow on AMD GPUs

> ssh wn-lot-001

> apptainer build --sandbox --fix-perms /data/my_group/my_username/tensorflow_rocm docker://rocm/tensorflow:latest

> apptainer run --rocm /data/my_group/my_username/tensorflow_rocm
Apptainer> python
Python 3.9.13 (main, May 23 2022, 22:01:06)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tensorflow.python.client import device_lib
>>> device_lib.list_local_devices()
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
incarnation: 6969131800811270632
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 33818673152
locality {
  bus_id: 2
  numa_node: 1
  links {
    link {
      device_id: 1
      type: "StreamExecutor"
      strength: 1
incarnation: 15068533579864484951
physical_device_desc: "device: 0, name: Vega 20, pci bus id: 0000:83:00.0"
, name: "/device:GPU:1"
device_type: "GPU"
memory_limit: 33818673152
locality {
  bus_id: 2
  numa_node: 1
  links {
    link {
      type: "StreamExecutor"
      strength: 1
incarnation: 16034552143571767030
physical_device_desc: "device: 1, name: Vega 20, pci bus id: 0000:43:00.0"


  • Email for help using containers.