Custom Environments for JupyterHub

Info

Interactive code interpreters which are used by Jupyter notebooks are called kernels. Creating and using your own kernel has the benefit, that you can install your own preferred Python packages and use them in your notebooks.

We currently have two different architectures at ZIH systems. Build your kernel environment on the same architecture that you want to use later on with the kernel. In the examples below, we use the name "my-kernel" for our user kernel. We recommend to prefix your kernels with keywords like alpha, barnard, romeo, power9, venv, conda. This way, you can later recognize easier how you built the kernel and on which hardware it will work. Depending on that hardware, allocate resources as follows.

Cluster	Architecture name
Alpha	x86_64 (AMD)
Barnard	x86_64 (Intel)
Capella	x86_64 (AMD)
Romeo	x86_64 (AMD)
Power9	ppc64le (IBM)

Preliminary Steps

Start an interactive job

```console
maria@login.<cluster>$ srun --pty --ntasks=1 --cpus-per-task=2 \
 --mem-per-cpu=2541 --time=02:00:00 bash -l
```

When creating a virtual environment in your home directory, you got to decide to either use "Python virtualenv" or "conda environment".

Note

Please keep in mind that Python virtualenv is the preferred way to create a Python virtual environment. For working with conda virtual environments, it may be necessary to configure your shell as described in Python virtual environments

Python Virtualenv

While we have a general description on Python Virtual Environments, here we have a more detailed description on using them with JupyterHub:

Depending on the Cluster that you are targeting, please choose the right modules:

release/23.10release/23.04

For use with Python version 3.10.4, please try to initialize your Python Virtual Environment like this:

[marie@barnard ~]$ module load release/23.10  GCC/11.3.0  Python/3.10.4
Module GCC/11.3.0, Python/3.10.4 and 12 dependencies loaded.

[marie@barnard ~]$ module load release/23.04  GCC/11.3.0  Python/3.10.4
Module GCC/11.3.0, Python/3.10.4 and 12 dependencies loaded.

Then continue with the steps below.

```console
[marie@barnard ~]$ mkdir -p ~/usr/jlab-kernels # please use workspaces!
[marie@barnard ~]$ cd ~/usr/jlab-kernels
[marie@barnard jlab-kernels]$ python3 -m venv --system-site-packages my-kernel
[marie@barnard jlab-kernels]$
[marie@barnard jlab-kernels]$ source my-kernel/bin/activate
(my-kernel) [marie@barnard jlab-kernels]$
(my-kernel) [marie@barnard jlab-kernels]$ pip install ipykernel
Collecting ipykernel
[...]
Successfully installed [...] ipykernel-x.x.x ipython-x.x.x [...]
```

After following the initialization of the environment (above), the usage of Python's Package manager pip is the same:

(my-kernel) marie@compute$ pip install --upgrade pip
(my-kernel) marie@compute$ python -m ipykernel install --user --name my-kernel --display-name="my kernel"
Installed kernelspec my-kernel in .../.local/share/jupyter/kernels/my-kernel
(my-kernel) marie@compute$ pip install [...] # now install additional packages for your notebooks
(my-kernel) marie@compute$ deactivate

Conda Environment

Load the needed module depending on Cluster architecture:

Nodes with x86_64 CPUNodes with ppc64le CPU

marie@compute$ module load Anaconda3

marie@ml$ module load PythonAnaconda

Hint

For working with conda virtual environments, it may be necessary to configure your shell as described in Python virtual environments.

Continue with environment creation, package installation and kernel registration:

marie@compute$ mkdir user-kernel # please use workspaces!
marie@compute$ conda create --prefix $HOME/user-kernel/my-kernel python=3.8.6
Collecting package metadata: done
Solving environment: done
[...]
marie@compute$ conda activate $HOME/user-kernel/my-kernel
marie@compute$ conda install ipykernel
Collecting package metadata: done
Solving environment: done
[...]
marie@compute$ python -m ipykernel install --user --name my-kernel --display-name="my kernel"
Installed kernelspec my-kernel in [...]
marie@compute$ conda install [..] # now install additional packages for your notebooks
marie@compute$ conda deactivate

Using your custom environment

Now you can start a new session and your kernel should be available.

JupyterLabClassic Jupyter notebook

Your kernels are listed on the launcher page:

You can switch kernels of existing notebooks in the menu:

Your kernel is listed in the New menu:

You can switch kernels of existing notebooks in the kernel menu:

Note

Both python venv and conda virtual environments will be mentioned in the same list.