Info alert:Important Notice

Please note that more information about the previous v2 releases can be found here. You can use "Find a release" search bar to search for a particular release.

Managing resources

Table of Contents

Creating custom workbench images
Managing cluster PVC size
- Configuring the default PVC size for your cluster
- Restoring the default PVC size for your cluster
Managing storage classes
- Configuring storage class settings
- Configuring the default storage class for your cluster
Selecting Open Data Hub administrator and user groups
Managing Jupyter notebook servers

As an Open Data Hub administrator, you can manage the following resources:

Custom workbench images
Cluster PVC size
Cluster storage classes
Open Data Hub admin and user groups
Jupyter notebook servers

Creating custom workbench images

Open Data Hub includes a selection of default workbench images that a data scientist can select when they create or edit a workbench.

In addition, you can import a custom workbench image, for example, if you want to add libraries that data scientists often use, or if your data scientists require a specific version of a library that is different from the version provided in a default image. Custom workbench images are also useful if your data scientists require operating system packages or applications because they cannot install them directly in their running environment (data scientist users don’t have root access, which is needed for those operations).

A custom workbench image is simply a container image. You build one as you would build any standard container image, by using a Containerfile (or Dockerfile). You start from an existing image (the FROM instruction), and then add your required elements.

You have the following options for creating a custom workbench image:

Start from one of the default images, as described in Creating a custom image from a default Open Data Hub image
Create your own image by following the guidelines for making it compatible with Open Data Hub, as described in Creating a custom image from your own image

Additional resources

For more information about creating images, see the following resources:

Creating a custom image from a default Open Data Hub image

After Open Data Hub is installed on a cluster, you can find the default workbench images in the OpenShift console, under Builds → ImageStreams for the redhat-ods-applications project.

You can create a custom image by adding OS packages or applications to a default Open Data Hub image.

Prerequisites

You know which default image you want to use as the base for your custom image.
You have cluster-admin access to the OpenShift console for the cluster where Open Data Hub is installed.

Procedure

Obtain the location of the default image that you want to use as the base for your custom image.
1. In the OpenShift console, select Builds → ImageStreams.
2. Select the redhat-ods-applications project.
3. From the list of installed imagestreams, click the name of the image that you want to use as the base for your custom image. For example, click pytorch.
4. On the ImageStream details page, click YAML.
5. In the spec:tags section, find the tag for the version of the image that you want to use.
  
  The location of the original image is shown in the tag’s from:name section, for example:
  
  name: 'quay.io/modh/odh-pytorch-notebook@sha256:b68e0192abf7d…'
6. Copy this location for use in your custom image.
Create a standard Containerfile or Dockerfile.
For the FROM instruction, specify the base image location that you copied in Step 1, for example:

FROM quay.io/modh/odh-pytorch-notebook@sha256:b68e0…

Optional: Install OS images:

Switch to USER 0 (USER 0 is required to install OS packages).
Install the packages.

Switch back to USER 1001.

The following example creates a custom workbench image that adds Java to the default PyTorch image:

 FROM quay.io/modh/odh-pytorch-notebook@sha256:b68e0…

 USER 0

 RUN INSTALL_PKGS="java-11-openjdk java-11-openjdk-devel" && \
    dnf install -y --setopt=tsflags=nodocs $INSTALL_PKGS && \
    dnf -y clean all --enablerepo=*

 USER 1001

Optional: Add Python packages:
1. Specify USER 1001.
2. Copy the requirements.txt file.
3. Install the packages.
  
  The following example installs packages from the requirements.txt file in the default PyTorch image:
  FROM quay.io/modh/odh-pytorch-notebook@sha256:b68e0… USER 1001 COPY requirements.txt ./requirements.txt RUN pip install -r requirements.txt
Build the image file. For example, you can use podman build locally where the image file is located and then push the image to a registry that is accessible to Open Data Hub:
```
$ podman build -t my-registry/my-custom-image:0.0.1 .
$ podman push my-registry/my-custom-image:0.0.1
```
Alternatively, you can leverage OpenShift’s image build capabilities by using BuildConfig.

Creating a custom image from your own image

You can build your own custom image. However, you must make sure that your image is compatible with OpenShift and Open Data Hub.

Additional resources

General Container image guidelines section in the OpenShift Container Platform Images documentation.
Red Hat Universal Base Image: https://catalog.redhat.com/software/base-images
Red Hat Ecosystem Catalog: https://catalog.redhat.com/

Basic guidelines for creating your own workbench image

The following basic guidelines provide information to consider when you build your own custom notebook image.

Designing your image to run with USER 1001

In OpenShift, your container will run with a random UID and a GID of 0. Make sure that your image is compatible with these user and group requirements, especially if you need write access to directories. Best practice is to design your image to run with USER 1001.

Avoid placing artifacts in $HOME

The persistent volume attached to the workbench will be mounted on /opt/app-root/src. This location is also the location of $HOME. Therefore, do not put any files or other resources directly in $HOME because they won’t be visible after the workbench is deployed (and the persistent volume is mounted).

Specifying the API endpoint

OpenShift readiness and liveness probes will query the /api endpoint. For a Jupyter IDE, this is the default endpoint. For other IDEs, you must implement the /api endpoint.

Advanced guidelines for creating your own workbench image

The following guidelines provide information to consider when you build your own custom notebook image.

Minimizing image size

A notebook image uses a "layered" file system. Every time you use a COPY or a RUN command in your notebook image file, a new layer is created. Artifacts are not deleted. When you remove an artifact, for example, a file, it is "masked" in the next layer. Therefore, consider the following guidelines when you create your notebook image file.

Avoid using the dnf update command.
- If you start from an image that is constantly updated, such as ubi9/python-39 from the Red Hat Catalog, you might not need to use the dnf update command. This command fetches new metadata, updates files that might not have impact, and increases the notebook image size.
- Point to a newer version of your base image rather than performing a dnf update on an older version.
Group RUN commands. Chain your commands by adding && \ at the end of each line.
If you must compile code (such as a library or an application) to include in your custom image, implement multi-stage builds so that you avoid including the build artifacts in your final image. That is, compile the library or application in an intermediate image and then copy the result to your final image, leaving behind build artifacts that you do not want included.

Setting access to files and directories

Set the ownership of files and folders to 1001:0 (user "default", group "0"), for example:
```
COPY --chown=1001:0 os-packages.txt ./
```
On OpenShift, every container is in a standard namespace (unless you modify security). The container runs with a user that has a random user ID (uid) and with a group ID (gid) of 0. Therefore, all folders that you want to write to - and all the files you want to (temporarily) modify - in your image must be accessible by the user that has the random user ID (uid). Alternatively, you can set access to any user, as shown in the following example:
```
COPY --chmod=775 os-packages.txt ./
```
Build your image with /opt/app-root/src as the default location for the data that you want persisted, for example:
```
WORKDIR /opt/app-root/src
```
When a user launches a workbench from the Open Data Hub Applications → Enabled page, the "personal" volume of the user is mounted at /opt/app-root/src. Because this location is not configurable, when you build your custom image, you must specify this default location for persisted data.
Fix permissions to support PIP (the package manager for Python packages) in OpenShift environments. Add the following command to your custom image (if needed, change python3.9 to the Python version that you are using):
```
chmod -R g+w /opt/app-root/lib/python3.9/site-packages && \
   fix-permissions /opt/app-root -P
```
A service within your notebook image must answer at ${NB_PREFIX}/api, otherwise the OpenShift liveness/readiness probes fail and delete the pod for the workbench image.

The NB_PREFIX environment variable specifies the URL path where the container is expected to be listening.

The following is an example of an Nginx configuration:
```
location = ${NB_PREFIX}/api {
	return 302  /healthz;
	access_log  off;
}
```

For idle culling to work, the ${NB_PREFIX}/api/kernels URL must return a specifically-formatted JSON payload, as shown in the following example:

The following is an example of an Nginx configuration:

location = ${NB_PREFIX}/api/kernels {
	return 302 $custom_scheme://$http_host/api/kernels/;
	access_log  off;
}

location ${NB_PREFIX}/api/kernels/ {
	return 302 $custom_scheme://$http_host/api/kernels/;
	access_log  off;
}

location /api/kernels/ {
  index access.cgi;
  fastcgi_index access.cgi;
  gzip  off;
  access_log	off;
 }

The returned JSON payload should be:

{"id":"rstudio","name":"rstudio","last_activity":(time in ISO8601 format),"execution_state":"busy","connections": 1}

Enabling CodeReady Builder (CRB) and Extra Packages for Enterprise Linux (EPEL)

CRB and EPEL are repositories that provide packages which are absent from a standard Red Hat Enterprise Linux (RHEL) or Universal Base Image (UBI) installation. They are useful and required for installing some software, for example, RStudio.

On UBI9 images, CRB is enabled by default. To enable EPEL on UBI9-based images, run the following command:

 RUN yum install -y https://download.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

To enable CRB and EPEL on Centos Stream 9-based images, run the following command:

 RUN yum install -y yum-utils && \
    yum-config-manager --enable crb && \
    yum install -y https://download.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

Adding Elyra compatibility

Support for data science pipelines V2 (provided with the odh-elyra package) is available in Open Data Hub version 2.9 and later. Previous versions of Open Data Hub support data science pipelines V1 (provided with the elyra package).

If you want your custom image to support data science pipelines V2, you must address the following requirements:

Include the odh-elyra package for having support with Data Science pipeline V2 (not the elyra package), for example:
```
 USER 1001

 RUN pip install odh-elyra
```
If you want to include the data science pipeline configuration automatically, as a runtime configuration, add an annotation when you import a custom workbench image.

Enabling custom images in Open Data Hub

All Open Data Hub admin users can import custom workbench images, by default, by selecting the Settings → Notebook images navigation option in the Open Data Hub dashboard.

If the Settings → Notebook images option is not available, check the following settings, depending on which navigation element does not appear in the dashboard:

The Settings menu does not appear in the Open Data Hub navigation bar

The visibility of the Open Data Hub dashboard’s Settings menu is determined by your user permissions. By default, the *Settings menu is available to Open Data Hub administration users (users that are members of the rhoai-admins group). Users with the OpenShift cluster-admin role, are automatically added to the rhoai-admins group and are granted administrator access in Open Data Hub.

For more information about user permissions, see Managing groups and users.
The Notebook images menu option does not appear under the Settings menu

The visibility of the Notebook images menu option is controlled in the dashboard configuration, by the value of the dashboardConfig: disableBYONImageStream option. It is set to false (the Notebook images menu option is visible) by default.

You need cluster-admin permissions to edit the dashboard configuration.

For more information about setting dashboard configuration options, see Customizing the dashboard.

Importing a custom workbench image

You can import custom workbench images that cater to your Open Data Hub project’s specific requirements. From the Notebook images page, you can enable or disable a previously imported workbench image and create an accelerator profile as a recommended accelerator for existing notebook images.

You must import it so that your Open Data Hub users (data scientists) can access it when they create a project workbench.

Prerequisites

You have logged in to Open Data Hub.
You have Open Data Hub administrator permissions.
Your custom image exists in an image registry that is accessible to Open Data Hub.
The Settings → Notebook images dashboard navigation menu option is enabled, as described in Creating a custom image from a default Open Data Hub image.
If you want to associate an accelerator with the custom image that you want to import, you know the accelerator’s identifier - the unique string that identifies the hardware accelerator.

Procedure

From the Open Data Hub dashboard, click Settings → Notebook images.

The Notebook images page appears. Previously imported images are displayed. To enable or disable a previously imported image, on the row containing the relevant image, click the toggle in the Enable column.
Optional: If you want to associate an accelerator and you have not already created an accelerator profile, click Create profile on the row containing the image and complete the relevant fields. If the image does not contain an accelerator identifier, you must manually configure one before creating an associated accelerator profile.
Click Import new image. Alternatively, if no previously imported images were found, click Import image.

The Import Notebook images dialog appears.
In the Image location field, enter the URL of the repository containing the image. For example: quay.io/my-repo/my-image:tag, quay.io/my-repo/my-image@sha256:xxxxxxxxxxxxx, or docker.io/my-repo/my-image:tag.
In the Name field, enter an appropriate name for the image.
Optional: In the Description field, enter a description for the image.
Optional: From the Accelerator identifier list, select an identifier to set its accelerator as recommended with the image. If the image contains only one accelerator identifier, the identifier name displays by default.
Optional: Add software to the image. After the import has completed, the software is added to the image’s meta-data and displayed on the Jupyter server creation page.
1. Click the Software tab.
2. Click the Add software button.
3. Click Edit ().
4. Enter the Software name.
5. Enter the software Version.
6. Click Confirm () to confirm your entry.
7. To add additional software, click Add software, complete the relevant fields, and confirm your entry.
Optional: Add packages to the notebook images. After the import has completed, the packages are added to the image’s meta-data and displayed on the Jupyter server creation page.
1. Click the Packages tab.
2. Click the Add package button.
3. Click Edit ().
4. Enter the Package name. For example, if you want to include data science pipeline V2 automatically, as a runtime configuration, type odh-elyra.
5. Enter the package Version. For example, type 3.16.7.
6. Click Confirm () to confirm your entry.
7. To add an additional package, click Add package, complete the relevant fields, and confirm your entry.
Click Import.

Verification

The image that you imported is displayed in the table on the Notebook images page.
Your custom image is available for selection when a user creates a workbench.

Additional resources

Managing cluster PVC size

Configuring the default PVC size for your cluster

To configure how resources are claimed within your Open Data Hub cluster, you can change the default size of the cluster’s persistent volume claim (PVC) ensuring that the storage requested matches your common storage workflow. PVCs are requests for resources in your cluster and also act as claim checks to the resource.

Prerequisites

You have logged in to Open Data Hub.

Note	Changing the PVC setting restarts the Jupyter pod and makes Jupyter unavailable for up to 30 seconds. As a workaround, it is recommended that you perform this action outside of your organization’s typical working day.

Procedure

From the Open Data Hub dashboard, click Settings → Cluster settings.
Under PVC size, enter a new size in gibibytes or mebibytes.
Click Save changes.

Verification

New PVCs are created with the default storage size that you configured.

Additional resources

Understanding persistent storage

Restoring the default PVC size for your cluster

To change the size of resources utilized within your Open Data Hub cluster, you can restore the default size of your cluster’s persistent volume claim (PVC).

Prerequisites

You have logged in to Open Data Hub.
You are part of the administrator group for Open Data Hub in OpenShift Container Platform.

Procedure

From the Open Data Hub dashboard, click Settings → Cluster settings.
Click Restore Default to restore the default PVC size of 20GiB.
Click Save changes.

Verification

New PVCs are created with the default storage size of 20 GiB.

Additional resources

Understanding persistent storage

Managing storage classes

OpenShift Container Platform cluster administrators use storage classes to describe the different types of storage that is available in their cluster. These storage types can represent different quality-of-service levels, backup policies, or other custom policies set by cluster administrators.

Configuring storage class settings

As an Open Data Hub administrator, you can manage OpenShift Container Platform cluster storage class settings for usage within Open Data Hub, including the display name, description, and whether users can use the storage class when creating or editing cluster storage. These settings do not impact the storage class within OpenShift Container Platform.

Prerequisites

You have logged in to Open Data Hub.
You have Open Data Hub administrator permissions.

Procedure

From the Open Data Hub dashboard, click Settings → Storage classes.

The Storage classes page appears, displaying the storage classes for your cluster as defined in OpenShift Container Platform.
To enable or disable a storage class for users, on the row containing the storage class, click the toggle in the Enable column.
To edit a storage class, on the row containing the storage class, click the action menu (⋮) and then select Edit.

The Edit storage class details dialog opens.
Optional: In the Display Name field, update the name for the storage class. This name is used only in Open Data Hub and does not impact the storage class within OpenShift Container Platform.
Optional: In the Description field, update the description for the storage class. This description is used only in Open Data Hub and does not impact the storage class within OpenShift Container Platform.
Click Save.

Verification

If you enabled a storage class, the storage class is available for selection when a user adds cluster storage to a data science project or workbench.
If you disabled a storage class, the storage class is not available for selection when a user adds cluster storage to a data science project or workbench.
If you edited a storage class name, the updated storage class name is displayed when a user adds cluster storage to a data science project or workbench.

Additional resources

Storage classes in OpenShift Container Platform

Configuring the default storage class for your cluster

As an Open Data Hub administrator, you can configure the default storage class for Open Data Hub to be different from the default storage class in OpenShift Container Platform.

Prerequisites

You have logged in to Open Data Hub.
You have Open Data Hub administrator permissions.

Procedure

From the Open Data Hub dashboard, click Settings → Storage classes.

The Storage classes page appears, displaying the storage classes for your cluster as defined in OpenShift Container Platform.
If the storage class that you want to set as the default is not enabled, on the row containing the storage class, click the toggle in the Enable column.
To set a storage class as the default for Open Data Hub, on the row containing the storage class, select Set as default.

Verification

When a user adds cluster storage to a data science project or workbench, the default storage class that you configured is automatically selected.

Additional resources

Storage classes in OpenShift Container Platform

Selecting Open Data Hub administrator and user groups

By default, all users authenticated in OpenShift can access Open Data Hub.

Also by default, users with cluster-admin permissions are Open Data Hub administrators. A cluster admin is a superuser that can perform any action in any project in the OpenShift cluster. When bound to a user with a local binding, they have full control over quota and every action on every resource in the project.

After a cluster admin user defines additional administrator and user groups in OpenShift, you can add those groups to Open Data Hub by selecting them in the Open Data Hub dashboard.

Prerequisites

You have administrator privileges in Open Data Hub and can view the Settings navigation option in the Open Data Hub dashboard.
You have logged in to Open Data Hub as described in Logging in to Open Data Hub.
The groups that you want to select as administrator and user groups for Open Data Hub already exist in OpenShift Container Platform. For more information, see Managing users and groups.

Procedure

From the Open Data Hub dashboard, click Settings → User management.
Select your Open Data Hub admin groups: Under Data science administrator groups, click the text box and select an OpenShift group. Repeat this process to define multiple admin groups.
Select your Open Data Hub user groups: Under Data science user groups, click the text box and select an OpenShift group. Repeat this process to define multiple user groups.

Important
The system:authenticated setting allows all users authenticated in OpenShift to access Open Data Hub.
Click Save changes.

Verification

Administrator users can successfully log in to Open Data Hub and have access to the Settings navigation menu.
Non-administrator users can successfully log in to Open Data Hub. They can also access and use individual components, such as projects and workbenches.

Managing Jupyter notebook servers

Accessing the Jupyter administration interface

You can use the Jupyter administration interface to control notebook servers in your Open Data Hub environment.

Prerequisite

You have OpenShift cluster-admin privileges.

Procedure

To access the Jupyter administration interface from Open Data Hub, perform the following actions:
1. In Open Data Hub, in the Applications section of the left menu, click Enabled.
2. Locate the Jupyter tile and click Launch application.
3. On the page that opens when you launch Jupyter, click the Administration tab.
  
  The Administration page opens.
To access the Jupyter administration interface from JupyterLab, perform the following actions:
1. Click File → Hub Control Panel.
2. On the page that opens in Open Data Hub, click the Administration tab.
  
  The Administration page opens.

Verification

You can see the Jupyter administration interface.

Starting notebook servers owned by other users

Administrators can start a notebook server for another existing user from the Jupyter administration interface.

Prerequisites

You are part of the OpenShift Container Platform administrator group which requires the cluster-admin role on OpenShift Container Platform. For more information, see Creating a cluster admin.
You have launched the Jupyter application, as described in Starting a Jupyter notebook server.

Procedure

On the page that opens when you launch Jupyter, click the Administration tab.
On the Administration tab, perform the following actions:
1. In the Users section, locate the user whose notebook server you want to start.
2. Click Start server beside the relevant user.
3. Complete the Start a notebook server page.
4. Optional: Select the Start server in current tab checkbox if necessary.
5. Click Start server.
  
  After the server starts, you see one of the following behaviors:
  If you previously selected the Start server in current tab checkbox, the JupyterLab interface opens in the current tab of your web browser.
  
  If you did not previously select the Start server in current tab checkbox, the Starting server dialog box prompts you to open the server in a new browser tab or in the current tab.
  
  The JupyterLab interface opens according to your selection.

Verification

The JupyterLab interface opens.

Accessing notebook servers owned by other users

Administrators can access notebook servers that are owned by other users to correct configuration errors or to help them troubleshoot problems with their environment.

Prerequisites

You have OpenShift cluster-admin privileges.
You have launched the Jupyter application, as described in Starting a Jupyter notebook server.
The notebook server that you want to access is running.

Procedure

On the page that opens when you launch Jupyter, click the Administration tab.
On the Administration page, perform the following actions:
1. In the Users section, locate the user that the notebook server belongs to.
2. Click View server beside the relevant user.
3. On the Notebook server control panel page, click Access notebook server.

Verification

The user’s notebook server opens in JupyterLab.

Stopping notebook servers owned by other users

Administrators can stop notebook servers that are owned by other users to reduce resource consumption on the cluster, or as part of removing a user and their resources from the cluster.

Prerequisites

If you are using Open Data Hub groups, you are part of the administrator group (for example, {oai-admin-group}). If you are not using groups, you have OpenShift cluster-admin privileges.
You have launched the Jupyter application, as described in Starting a Jupyter notebook server.
The notebook server that you want to stop is running.

Procedure

On the page that opens when you launch Jupyter, click the Administration tab.
Stop one or more servers.
- If you want to stop one or more specific servers, perform the following actions:
  1. In the Users section, locate the user that the notebook server belongs to.
  2. To stop the notebook server, perform one of the following actions:
    
    Click the action menu (⋮) beside the relevant user and select Stop server.
    
    Click View server beside the relevant user and then click Stop notebook server.
    
    The Stop server dialog box appears.
  3. Click Stop server.
- If you want to stop all servers, perform the following actions:
  1. Click the Stop all servers button.
  2. Click OK to confirm stopping all servers.

Verification

The Stop server link beside each server changes to a Start server link when the notebook server has stopped.

Stopping idle notebooks

You can reduce resource usage in your Open Data Hub deployment by stopping notebook servers that have been idle (without logged in users) for a period of time. This is useful when resource demand in the cluster is high. By default, idle notebooks are not stopped after a specific time limit.

Note	If you have configured your cluster settings to disconnect all users from a cluster after a specified time limit, then this setting takes precedence over the idle notebook time limit. Users are logged out of the cluster when their session duration reaches the cluster-wide time limit.

Prerequisites

You have logged in to Open Data Hub.
You are part of the administrator group for Open Data Hub in OpenShift Container Platform.

Procedure

From the Open Data Hub dashboard, click Settings → Cluster settings.
Under Stop idle notebooks, select Stop idle notebooks after.
Enter a time limit, in hours and minutes, for when idle notebooks are stopped.
Click Save changes.

Verification

The notebook-controller-culler-config ConfigMap, located in the redhat-ods-applications project on the Workloads → ConfigMaps page, contains the following culling configuration settings:
- ENABLE_CULLING: Specifies if the culling feature is enabled or disabled (this is false by default).
- IDLENESS_CHECK_PERIOD: The polling frequency to check for a notebook’s last known activity (in minutes).
- CULL_IDLE_TIME: The maximum allotted time to scale an inactive notebook to zero (in minutes).
Idle notebooks stop at the time limit that you set.

Adding notebook pod tolerations

If you want to dedicate certain machine pools to only running notebook pods, you can allow notebook pods to be scheduled on specific nodes by adding a toleration. Taints and tolerations allow a node to control which pods should (or should not) be scheduled on them. For more information, see Understanding taints and tolerations.

This capability is useful if you want to make sure that notebook servers are placed on nodes that can handle their needs. By preventing other workloads from running on these specific nodes, you can ensure that the necessary resources are available to users who need to work with large notebook sizes.

Prerequisites

You have logged in to Open Data Hub.
You are part of the administrator group for Open Data Hub in OpenShift Container Platform.
You are familiar with OpenShift Container Platform taints and tolerations, as described in Understanding taints and tolerations.

Procedure

From the Open Data Hub dashboard, click Settings → Cluster settings.
Under Notebook pod tolerations, select Add a toleration to notebook pods to allow them to be scheduled to tainted nodes.
In the Toleration key for notebook pods field, enter a toleration key. The key is any string, up to 253 characters. The key must begin with a letter or number, and can contain letters, numbers, hyphens, dots, and underscores. For example, notebooks-only.
Click Save changes. The toleration key is applied to new notebook pods when they are created.

For existing notebook pods, the toleration key is applied when the notebook pods are restarted. If you are using Jupyter, see Updating notebook server settings by restarting your server. If you are using a workbench in a data science project, see Starting a workbench.

Next step

In OpenShift Container Platform, add a matching taint key (with any value) to the machine pools that you want to dedicate to notebooks. For more information, see Controlling pod placement using node taints.

Verification

In the OpenShift Container Platform console, for a pod that is running, click Workloads → Pods. Otherwise, for a pod that is stopped, click Workloads → StatefulSet.
Search for your workbench pod name and then click the name to open the pod details page.
Confirm that the assigned Node and Tolerations are correct.

Troubleshooting common problems in Jupyter for administrators

If your users are experiencing errors in Open Data Hub relating to Jupyter, their notebooks, or their notebook server, read this section to understand what could be causing the problem, and how to resolve the problem.

A user receives a 404: Page not found error when logging in to Jupyter

Problem

If you have configured specialized user groups for Open Data Hub, the user name might not be added to the default user group for Open Data Hub.

Diagnosis

Check whether the user is part of the default user group.

Find the names of groups allowed access to Jupyter.
1. Log in to the OpenShift Container Platform web console.
2. Click User Management → Groups.
3. Click the name of your user group, for example, {oai-user-group}.
  
  The Group details page for that group appears.
Click the Details tab for the group and confirm that the Users section for the relevant group contains the users who have permission to access Jupyter.

Resolution

If the user is not added to any of the groups allowed access to Jupyter, add them.

A user’s notebook server does not start

Problem

The OpenShift Container Platform cluster that hosts the user’s notebook server might not have access to enough resources, or the Jupyter pod may have failed.

Diagnosis

Log in to the OpenShift Container Platform web console.
Delete and restart the notebook server pod for this user.
1. Click Workloads → Pods and set the Project to rhods-notebooks.
2. Search for the notebook server pod that belongs to this user, for example, jupyter-nb-<username>-*.
  
  If the notebook server pod exists, an intermittent failure may have occurred in the notebook server pod.
  
  If the notebook server pod for the user does not exist, continue with diagnosis.
Check the resources currently available in the OpenShift Container Platform cluster against the resources required by the selected notebook server image.

If worker nodes with sufficient CPU and RAM are available for scheduling in the cluster, continue with diagnosis.
Check the state of the Jupyter pod.

Resolution

If there was an intermittent failure of the notebook server pod:
1. Delete the notebook server pod that belongs to the user.
2. Ask the user to start their notebook server again.
If the notebook server does not have sufficient resources to run the selected notebook server image, either add more resources to the OpenShift Container Platform cluster, or choose a smaller image size.

The user receives a database or disk is full error or a no space left on device error when they run notebook cells

Problem

The user might have run out of storage space on their notebook server.

Diagnosis

Log in to Jupyter and start the notebook server that belongs to the user having problems. If the notebook server does not start, follow these steps to check whether the user has run out of storage space:
1. Log in to the OpenShift Container Platform web console.
2. Click Workloads → Pods and set the Project to rhods-notebooks.
3. Click the notebook server pod that belongs to this user, for example, jupyter-nb-<idp>-<username>-*.
4. Click Logs. The user has exceeded their available capacity if you see lines similar to the following:
  Unexpected error while saving file: XXXX database or disk is full

Resolution

Increase the user’s available storage by expanding their persistent volume.
Work with the user to identify files that can be deleted from the /opt/app-root/src directory on their notebook server to free up their existing storage space.

Note	When you delete files using the JupyterLab file explorer, the files move to the hidden `/opt/app-root/src/.local/share/Trash/files` folder in the persistent storage for the notebook. To free up storage space for notebooks, you must permanently delete these files.

QUICK LINKS

STAY IN TOUCH