Skip to content

JupyterHub

Aim: Provide information about how to use containers on Nikhef's computing infrastructure to run different operating systems.

Target audience: Users of the Stoomboot cluster.

Nikhef's JupyterHub service runs on a server with an HTTP proxy service so multiple users can run JupyterLab sessions on the same server&mdashwithout having to install any software on a laptop or desktop. JupyterLab, the next generation of Jupyter Notebooks, is a popular interactive Python notebook manager.

Prerequisites

  • A Nikhef account;
  • Access to the Nikhef network via eduVPN or being onsite;
  • An ssh client.

Usage

Accessing the service

Nikhef's JupyterHub service can be found at https://callysto.nikhef.nl. Note this is running an Ubuntu OS so the command line interface will look different.

It is available exclusively for Nikhef users. You have to be on the Nikhef network or use eduVPN to access the service.

Navigate to https://callysto.nikhef.nl and log on via Nikhef SSO.

A JupyterLab session will run under your Nikhef account with full access to your home directory, /project and /data shares, and Stoomboot's dCache storage. However, in the file directory from the JupyterHub page, you will only be able to see what is present in your home directory. To view files, or directories, in /data, /project in the GUI, this will require a symlink. From callysto's launcher GUI, select the terminal tab and create a symlink by entering commands like: 

## ln -s [path-to-automounted-dir] [new dir name to appear in home directory]
ln -s /data/datagrid/hester data-hester
ln -s /project/datagrid/hester project-hester
using your own project and name.

Setting up new environments

It is possible to set up additional kernels and environments with different Python versions or software suites.

The command-line examples below can be used in JupyterHub by starting a terminal from the launcher.

Creating environments and kernels using conda

Note creating new conda environments and kernels should be done from the /data/[your_group]/[username] directory to prevent running out of disk space. By default, conda will use $HOME/.conda/envs/... to store information about the environment and $HOME/.local/share/jupyter/kernels/.... This can fill up quickly. (See more information about setting up conda environments)

Add the new conda environments dir to your conda configuration with, for example: 

source /opt/tljh/user/etc/profile.d/conda.sh
conda config --append envs_dirs /data/[myfirstproject]/conda/envs
Check that this worked with 
conda config --get envs_dirs
or the contents of 
cat $HOME/.condarc
To create a new conda environment in /data, you need to specify the whole path: 
### Source a script you want to use (sometimes this is in your experiment’s CVMFS repository)
source /opt/tljh/user/etc/profile.d/conda.sh

### Create a conda environment with a name of your choice
conda create --prefix /data/myfirstproject/conda/envs/shared_python38 --name shared_python38 python=3.8

### Activate the conda environment with a name used above
conda activate shared_python38

### Install the ipython kernel — this is the python execution backend for JupyterLabs/notebooks
conda install ipykernel

### Run the ipykernel install from the kernel’s env pointing to the Jupyter environment.
python -m ipykernel install --user --name shared_python38 --display-name 'Python 3.8 (shared)'
Note that the name given to the kernel can be (but doesn't have to be) different from the name of the conda environment. From the JupyterLab terminal window, check if the conda env is active/ready with 
conda list -n python38

And/or check the kernel list with 

jupyter kernelspec list

Conda ROOT shared environment and kernel

Some environments require the setting of environment variables like PATH. A popular example is ROOT.

Setting up a ROOT environment requires the conda-forge channel. See the documentation at CERN about how to set it up.

source /opt/tljh/user/etc/profile.d/conda.sh
conda config --set channel_priority strict
conda create -c conda-forge --prefix /project/<my-project>/.../sharedrootenv root
After activating the environment in the terminal, create a new kernel 
python -m ipykernel install --user --name shared_root --display-name 'ROOT (shared)' --env PATH $PATH
The PATH setting is important because ROOT needs to find its binaries when the kernel is loaded.

The new ROOT kernel should show up in the JupyterLab launcher.

Creating environments and kernels using python venv

Creating a virtual env should also be done from the /data/[your_group]/[username] directory. Some example commands for setting this up are: 

python -m venv testing123
source testing123/bin/activate
pip install ipykernel
python -m ipykernel install --user --name testing123 --display-name 'Testing 1,2,3'
This creates a new python venv in $PWD/testing123 and installs the Jupyter kernel in $HOME/.local/share/jupyter/kernels/testing123. List/remove/etc the kernel 
jupyter-kernelspec help
jupyter-kernelspec list
jupyter-kernelspec remove testing123
Removing environments conda based: 
conda env list
conda env remove -n python38
remove the previously created testing123 python environment directory and its contents with 
"rm -rf ${path_to_testing123}"

Contact

  • Email stbc-users@nikhef.nl or join the stbc-users on MatterMost for help using JupyterHub.