Dask

Aim: Provide the basics of how to use Dask on the Stoomboot cluster.

Introduction

Dask is a flexible open-source Python library for parallel computing, with dask_jobqueue you can scale your jobs across a cluster if the cluster runs a supported scheduling system. Dask supports HTcondor wat we use for stoomboot.

Python example

However there are some limitations in regards of Dask on stoomboot, given the scheduler and the workers need to be able to communicate with each other you need to explicitly specify the ports that are in use in a range that is open in the firewall. For the current stoomboot cluster this is 20000-25000. Given multiple workers can land on the same node hardcoding port does not work but we can use the features condor has in the submision language to work around this.

import random

cluster = HTCondorCluster(
        cores=1,
        memory="4GB",
        disk="2GB",
        job_extra_directives={
            "+UseOS": "\"el9\"",
            "+JobCategory": "\"short\""
        },
        scheduler_options={"port":random.randint(21000,25000)},
        worker_extra_args=["--worker-port=$RANDOM_INTEGER(21000,25000)"],
        shared_temp_directory="/data/<your data dir>",
)

As seen the jobCatagory and UseOS are specified and need extra \" escaped. Other then that we pick a random port for the scheduler and for each worker that gets submitted we randomise the worker port too. You can use ssh port forwarding to access the dask daskboard, this is also explained in the Dask documenation.

Known issues

Be aware tho on the interactive nodes the binary python exists however inside the containers only python3 exists so make sure when you create your venv on your host system that you use python3