Jupyter Notebook on the HPC

How to run a jupyter notebook instance on the HPC

Introduction

Jupyter Notebooks are often used in computational research and data science projects. Here’s how to set it up on the HPC:

---

(Recommended) Creating a Python Virtual Environment

Creating a virtual environment before running a jupyter-notebook instance is recommended for module cleanliness–in addition, it is very simple to create one!

In the terminal, type `python -m venv [your desired directory]. The picture below shows an example directory you could try:

Now, cd into your folder. You should see something like this:

Now, we must install the required dependencies(in this case, jupyter). In the terminal, simply run pip3 install jupyter to install the required modules.

Finally, we’ll kickstart this jupyter instance through running a .sh file with sbatch. Create a new file to submit to the cluster, and make sure to include ~/[directory]/bin/activate, as well as module load python/[your_version].

Now, type in jupyter notebook into the file. save it, then submit it with sbatch. For context, this is an example:

#!/bin/bash
#SBATCH --cpus-per-task=1
#SBATCH --mem=10G
#SBATCH --time=1-00:00:00
#SBATCH --gres=gpu:1
#SBATCH --partition=agpuq

#module load nvhpc/23.11
#module load cuda/12.3
module load python/3.12.1

. ~/pythonGPU/bin/activate

echo "*** Starting Jupyter on: "$(hostname)
jupyter notebook

Once running, you should see a new .out file. Open it up, and you should see something like this:

Accessing your Jupyter Notebook

Now that we have the instance running, we need to be able to view it on a browser tab, since if you try to use the link on a normal browser, it will not work, since we need specialized access.

The most convenient way to view your instances is through connecting to the cluster with X2go. To install X2go, simply download the version applicable to your device from this link. Once you do that, open up the application, and enter in the required parameters similar to the picture below. This is for the new session we will create. Make SURE to change the bottom parameter from the initial KDE to the XFCE option.

Pro tip: Notice that the host name is bigblue1.memphis.edu, and not bigblue.memphis.edu? This is because we actually have two login nodes–and they are represented by the corresponding numbers 1 and 2. If you wish to run both the virtual desktop session AND a terminal session at the same time, check for the last two digits after the @. If they say 01, make the host bigblue2.memphis.edu, and vice versa. This is optional, not required.

Now, continue by pressing OK.

You should see something like this, where the box on the right represents the session you want to start. Click on it.

Now, enter in your credentials, and you should see a window open, looking like a virtual desktop.

On the bottom bar, click on the browser option. There, a web browser should open. From there, paste the jupyter link the .out file gave when we submitted that batch script. From there, you should be able to see your jupyter notebook running!

Don’t forget to close your jupyter notebook instance when you are done. You can do this by running scancel [your job id] in the terminal.

---