Open Data Hub logo
DOCS
BLOG
COMMUNITY
GITHUB
GET STARTED

Info alert:Important Notice

Please note that more information about the previous v2 releases can be found here. You can use "Find a release" search bar to search for a particular release.

Working with accelerators

Use accelerators, such as NVIDIA GPUs, AMD GPUs, and Intel Gaudi AI accelerators, to optimize the performance of your end-to-end data science workflows.

Overview of accelerators

If you work with large data sets, you can use accelerators to optimize the performance of your data science models in Open Data Hub. With accelerators, you can scale your work, reduce latency, and increase productivity. You can use accelerators in Open Data Hub to assist your data scientists in the following tasks:

  • Natural language processing (NLP)

  • Inference

  • Training deep neural networks

  • Data cleansing and data processing

Open Data Hub supports the following accelerators:

  • NVIDIA graphics processing units (GPUs)

    • To use compute-heavy workloads in your models, you can enable NVIDIA graphics processing units (GPUs) in Open Data Hub.

    • To enable NVIDIA GPUs on OpenShift, you must install the NVIDIA GPU Operator.

  • AMD graphics processing units (GPUs)

    • Use the AMD GPU Operator to enable AMD GPUs for workloads such as AI/ML training and inference.

    • To enable AMD GPUs on OpenShift, you must do the following tasks:

    • Once installed, the AMD GPU Operator allows you to use the ROCm workbench images to streamline AI/ML workflows on AMD GPUs.

  • Intel Gaudi AI accelerators

    • Intel provides hardware accelerators intended for deep learning workloads.

    • Before you can enable Intel Gaudi AI accelerators in Open Data Hub, you must install the necessary dependencies. Also, the version of the Intel Gaudi AI Operator that you install must match the version of the corresponding workbench image in your deployment.

    • A workbench image for Intel Gaudi accelerators is not included in Open Data Hub by default. Instead, you must create and configure a custom notebook to enable Intel Gaudi AI support.

    • You can enable Intel Gaudi AI accelerators on-premises or with AWS DL1 compute nodes on an AWS instance.

  • Before you can use an accelerator in Open Data Hub, you must enable GPU support in Open Data Hub. This includes installing the Node Feature Discovery and NVIDIA GPU Operators. For more information, see NVIDIA GPU Operator on Red Hat OpenShift Container Platform in the NVIDIA documentation. In addition, your OpenShift instance must contain an associated accelerator profile. For accelerators that are new to your deployment, you must configure an accelerator profile for the accelerator in context. You can create an accelerator profile from the Settings → Accelerator profiles page on the Open Data Hub dashboard. If your deployment contains existing accelerators that had associated accelerator profiles already configured, an accelerator profile is automatically created after you upgrade to the latest version of Open Data Hub.

Additional resources

Enabling NVIDIA GPUs

Before you can use NVIDIA GPUs in Open Data Hub, you must install the NVIDIA GPU Operator.

Prerequisites

  • You have logged in to your OpenShift Container Platform cluster.

  • You have the cluster-admin role in your OpenShift Container Platform cluster.

  • You have installed an NVIDIA GPU and confirmed that it is detected in your environment.

Procedure

  1. To enable GPU support on an OpenShift cluster, follow the instructions here: NVIDIA GPU Operator on Red Hat OpenShift Container Platform in the NVIDIA documentation.

    Important

    After you install the Node Feature Discovery (NFD) Operator, you must create an instance of NodeFeatureDiscovery. In addition, after you install the NVIDIA GPU Operator, you must create a ClusterPolicy and populate it with default values.

  2. Delete the migration-gpu-status ConfigMap.

    1. In the OpenShift Container Platform web console, switch to the Administrator perspective.

    2. Set the Project to All Projects or redhat-ods-applications to ensure you can see the appropriate ConfigMap.

    3. Search for the migration-gpu-status ConfigMap.

    4. Click the action menu (⋮) and select Delete ConfigMap from the list.

      The Delete ConfigMap dialog appears.

    5. Inspect the dialog and confirm that you are deleting the correct ConfigMap.

    6. Click Delete.

  3. Restart the dashboard replicaset.

    1. In the OpenShift Container Platform web console, switch to the Administrator perspective.

    2. Click Workloads → Deployments.

    3. Set the Project to All Projects or redhat-ods-applications to ensure you can see the appropriate deployment.

    4. Search for the rhods-dashboard deployment.

    5. Click the action menu (⋮) and select Restart Rollout from the list.

    6. Wait until the Status column indicates that all pods in the rollout have fully restarted.

Verification

  • The reset migration-gpu-status instance is present on the Instances tab on the AcceleratorProfile custom resource definition (CRD) details page.

  • From the Administrator perspective, go to the Operators → Installed Operators page. Confirm that the following Operators appear:

    • NVIDIA GPU

    • Node Feature Discovery (NFD)

    • Kernel Module Management (KMM)

  • The GPU is correctly detected a few minutes after full installation of the Node Feature Discovery (NFD) and NVIDIA GPU Operators. The OpenShift Container Platform command line interface (CLI) displays the appropriate output for the GPU worker node. For example:

    # Expected output when the GPU is detected properly
oc describe node <node name>
...
Capacity:
  cpu:                4
  ephemeral-storage:  313981932Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             16076568Ki
  nvidia.com/gpu:     1
  pods:               250
Allocatable:
  cpu:                3920m
  ephemeral-storage:  288292006229
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             12828440Ki
  nvidia.com/gpu:     1
  pods:               250

After installing the NVIDIA GPU Operator, create an accelerator profile as described in Working with accelerators.

Intel Gaudi AI Accelerator integration

To accelerate your high-performance deep learning models, you can integrate Intel Gaudi AI accelerators into Open Data Hub. This integration enables your data scientists to use Gaudi libraries and software associated with Intel Gaudi AI accelerators through custom-configured workbench instances.

Intel Gaudi AI accelerators offer optimized performance for deep learning workloads, with the latest Gaudi 3 devices providing significant improvements in training speed and energy efficiency. These accelerators are suitable for enterprises running machine learning and AI applications on Open Data Hub.

Before you can enable Intel Gaudi AI accelerators in Open Data Hub, you must complete the following steps:

  1. Install the latest version of the Intel Gaudi AI Accelerator Operator from OperatorHub.

  2. Create and configure a custom workbench image for Intel Gaudi AI accelerators. A prebuilt workbench image for Gaudi accelerators is not included in Open Data Hub.

  3. Manually define and configure an accelerator profile for each Intel Gaudi AI device in your environment.

Open Data Hub supports Intel Gaudi devices up to Intel Gaudi 3. The Intel Gaudi 3 accelerators, in particular, offer the following benefits:

  • Improved training throughput: Reduce the time required to train large models by using advanced tensor processing cores and increased memory bandwidth.

  • Energy efficiency: Lower power consumption while maintaining high performance, reducing operational costs for large-scale deployments.

  • Scalable architecture: Scale across multiple nodes for distributed training configurations.

Your OpenShift platform must support EC2 DL1 instances to use Intel Gaudi AI accelerators in an Amazon EC2 DL1 instance. You can use Intel Gaudi AI accelerators in workbench instances or model serving after you enable the accelerators, create a custom workbench image, and configure the accelerator profile.

To identify the Intel Gaudi AI accelerators present in your deployment, use the lspci utility. For more information, see lspci(8) - Linux man page.

Important

The presence of Intel Gaudi AI accelerators in your deployment, as indicated by the lspci utility, does not guarantee that the devices are ready to use. You must ensure that all installation and configuration steps are completed successfully.
Additional resources

AMD GPU Integration

You can use AMD GPUs with Open Data Hub to accelerate AI and machine learning (ML) workloads. AMD GPUs provide high-performance compute capabilities, allowing users to process large data sets, train deep neural networks, and perform complex inference tasks more efficiently.

Integrating AMD GPUs with Open Data Hub involves the following components:

  • ROCm workbench images: Use the ROCm workbench images to streamline AI/ML workflows on AMD GPUs. These images include libraries and frameworks optimized with the AMD ROCm platform, enabling high-performance workloads for PyTorch and TensorFlow. The pre-configured images reduce setup time and provide an optimized environment for GPU-accelerated development and experimentation.

  • AMD GPU Operator: The AMD GPU Operator simplifies GPU integration by automating driver installation, device plugin setup, and node labeling for GPU resource management. It ensures compatibility between OpenShift and AMD hardware while enabling scaling of GPU-enabled workloads.

Verifying AMD GPU availability on your cluster

Before you proceed with the AMD GPU Operator installation process, you can verify the presence of an AMD GPU device on a node within your OpenShift Container Platform cluster. You can use commands such as lspci or oc to confirm hardware and resource availability.

Prerequisites

  • You have administrative access to the OpenShift Container Platform cluster.

  • You have a running OpenShift Container Platform cluster with a node equipped with an AMD GPU.

  • You have access to the OpenShift CLI (oc) and terminal access to the node.

Procedure

  1. Use the OpenShift CLI to verify if GPU resources are allocatable:

    1. List all nodes in the cluster to identify the node with an AMD GPU:

      oc get nodes

    2. Note the name of the node where you expect the AMD GPU to be present.

    3. Describe the node to check its resource allocation:

      oc describe node <node_name>

    4. In the output, locate the Capacity and Allocatable sections and confirm that amd.com/gpu is listed. For example:

      Capacity:
  amd.com/gpu:  1
Allocatable:
  amd.com/gpu:  1

  2. Check for the AMD GPU device using the lspci command:

    1. Log in to the node:

      oc debug node/<node_name>
chroot /host

    2. Run the lspci command and search for the supported AMD device in your deployment. For example:

      lspci | grep -E "MI210|MI250|MI300"

    3. Verify that the output includes one of the AMD GPU models. For example:

      03:00.0 Display controller: Advanced Micro Devices, Inc. [AMD] Instinct MI210

  3. Optional: Use the rocminfo command if the ROCm stack is installed on the node:

    rocminfo

    1. Confirm that the ROCm tool outputs details about the AMD GPU, such as compute units, memory, and driver status.

Verification

  • The oc describe node <node_name> command lists amd.com/gpu under Capacity and Allocatable.

  • The lspci command output identifies an AMD GPU as a PCI device matching one of the specified models (for example, MI210, MI250, MI300).

  • Optional: The rocminfo tool provides detailed GPU information, confirming driver and hardware configuration.

Additional resources

Enabling AMD GPUs

Before you can use AMD GPUs in Open Data Hub, you must install the required dependencies, deploy the AMD GPU Operator, and configure the environment.

Prerequisites

  • You have logged in to OpenShift Container Platform.

  • You have the cluster-admin role in OpenShift Container Platform.

  • You have installed your AMD GPU and confirmed that it is detected in your environment.

  • Your OpenShift Container Platform environment supports EC2 DL1 instances if you are running on Amazon Web Services (AWS).

Procedure

  1. Install the latest version of the AMD GPU Operator, as described in Install AMD GPU Operator on OpenShift.

  2. After installing the AMD GPU Operator, configure the AMD drivers required by the Operator as described in the documentation: Configure AMD drivers for the GPU Operator.

Note

Alternatively, you can install the AMD GPU Operator from the Red Hat Catalog. For more information, see Install AMD GPU Operator from Red Hat Catalog.

  1. After installing the AMD GPU Operator, create an accelerator profile, as described in Working with accelerator profiles.

Verification

From the Administrator perspective, go to the Operators → Installed Operators page. Confirm that the following Operators appear:

  • AMD GPU Operator

  • Node Feature Discovery (NFD)

  • Kernel Module Management (KMM)

Note

Ensure that you follow all the steps for proper driver installation and configuration. Incorrect installation or configuration may prevent the AMD GPUs from being recognized or functioning properly.

Working with accelerator profiles

To configure accelerators for your data scientists to use in Open Data Hub, you must create an associated accelerator profile. An accelerator profile is a custom resource definition (CRD) on OpenShift that has an AcceleratorProfile resource, and defines the specification of the accelerator. You can create and manage accelerator profiles by selecting Settings → Accelerator profiles on the Open Data Hub dashboard.

For accelerators that are new to your deployment, you must manually configure an accelerator profile for each accelerator. If your deployment contains an accelerator before you upgrade, the associated accelerator profile remains after the upgrade. You can manage the accelerators that appear to your data scientists by assigning specific accelerator profiles to your custom notebook images. This example shows the code for a Habana Gaudi 1 accelerator profile:

---
apiVersion: dashboard.opendatahub.io/v1alpha
kind: AcceleratorProfile
metadata:
  name: hpu-profile-first-gen-gaudi
spec:
  displayName: Habana HPU - 1st Gen Gaudi
  description: First Generation Habana Gaudi device
  enabled: true
  identifier: habana.ai/gaudi
  tolerations:
    - effect: NoSchedule
      key: habana.ai/gaudi
      operator: Exists
---

The accelerator profile code appears on the Instances tab on the details page for the AcceleratorProfile custom resource definition (CRD). For more information about accelerator profile attributes, see the following table:

Table 1. Accelerator profile attributes
Attribute Type Required Description

displayName

String

Required

The display name of the accelerator profile.

description

String

Optional

Descriptive text defining the accelerator profile.

identifier

String

Required

A unique identifier defining the accelerator resource.

enabled

Boolean

Required

Determines if the accelerator is visible in Open Data Hub.

tolerations

Array

Optional

The tolerations that can apply to notebooks and serving runtimes that use the accelerator. For more information about the toleration attributes that Open Data Hub supports, see Toleration v1 core.
Additional resources

Creating an accelerator profile

To configure accelerators for your data scientists to use in Open Data Hub, you must create an associated accelerator profile.

Prerequisites

  • You have logged in to Open Data Hub as a user with Open Data Hub administrator privileges.

Procedure

  1. From the Open Data Hub dashboard, click Settings → Accelerator profiles.

    The Accelerator profiles page appears, displaying existing accelerator profiles. To enable or disable an existing accelerator profile, on the row containing the relevant accelerator profile, click the toggle in the Enable column.

  2. Click Create accelerator profile.

    The Create accelerator profile dialog appears.

  3. In the Name field, enter a name for the accelerator profile.

  4. In the Identifier field, enter a unique string that identifies the hardware accelerator associated with the accelerator profile.

  5. Optional: In the Description field, enter a description for the accelerator profile.

  6. To enable or disable the accelerator profile immediately after creation, click the toggle in the Enable column.

  7. Optional: Add a toleration to schedule pods with matching taints.

    1. Click Add toleration.

      The Add toleration dialog opens.

    2. From the Operator list, select one of the following options:

      • Equal - The key/value/effect parameters must match. This is the default.

      • Exists - The key/effect parameters must match. You must leave a blank value parameter, which matches any.

    3. From the Effect list, select one of the following options:

      • None

      • NoSchedule - New pods that do not match the taint are not scheduled onto that node. Existing pods on the node remain.

      • PreferNoSchedule - New pods that do not match the taint might be scheduled onto that node, but the scheduler tries not to. Existing pods on the node remain.

      • NoExecute - New pods that do not match the taint cannot be scheduled onto that node. Existing pods on the node that do not have a matching toleration are removed.

    4. In the Key field, enter a toleration key. The key is any string, up to 253 characters. The key must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores.

    5. In the Value field, enter a toleration value. The value is any string, up to 63 characters. The value must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores.

    6. In the Toleration Seconds section, select one of the following options to specify how long a pod stays bound to a node that has a node condition.

      • Forever - Pods stays permanently bound to a node.

      • Custom value - Enter a value, in seconds, to define how long pods stay bound to a node that has a node condition.

    7. Click Add.

  8. Click Create accelerator profile.

Verification

  • The accelerator profile appears on the Accelerator profiles page.

  • The Accelerator list appears on the Start a notebook server page. After you select an accelerator, the Number of accelerators field appears, which you can use to choose the number of accelerators for your notebook server.

  • The accelerator profile appears on the Instances tab on the details page for the AcceleratorProfile custom resource definition (CRD).

Additional resources

Updating an accelerator profile

You can update the existing accelerator profiles in your deployment. You might want to change important identifying information, such as the display name, the identifier, or the description.

Prerequisites

  • You have logged in to Open Data Hub as a user with Open Data Hub administrator privileges.

  • The accelerator profile exists in your deployment.

Procedure

  1. From the Open Data Hub dashboard, click Settings → Notebook images.

    The Notebook images page appears. Previously imported notebook images are displayed. To enable or disable a previously imported notebook image, on the row containing the relevant notebook image, click the toggle in the Enable column.

  2. Click the action menu (⋮) and select Edit from the list.

    The Edit accelerator profile dialog opens.

  3. In the Name field, update the accelerator profile name.

  4. In the Identifier field, update the unique string that identifies the hardware accelerator associated with the accelerator profile, if applicable.

  5. Optional: In the Description field, update the accelerator profile.

  6. To enable or disable the accelerator profile immediately after creation, click the toggle in the Enable column.

  7. Optional: Add a toleration to schedule pods with matching taints.

    1. Click Add toleration.

      The Add toleration dialog opens.

    2. From the Operator list, select one of the following options:

      • Equal - The key/value/effect parameters must match. This is the default.

      • Exists - The key/effect parameters must match. You must leave a blank value parameter, which matches any.

    3. From the Effect list, select one of the following options:

      • None

      • NoSchedule - New pods that do not match the taint are not scheduled onto that node. Existing pods on the node remain.

      • PreferNoSchedule - New pods that do not match the taint might be scheduled onto that node, but the scheduler tries not to. Existing pods on the node remain.

      • NoExecute - New pods that do not match the taint cannot be scheduled onto that node. Existing pods on the node that do not have a matching toleration are removed.

    4. In the Key field, enter a toleration key. The key is any string, up to 253 characters. The key must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores.

    5. In the Value field, enter a toleration value. The value is any string, up to 63 characters. The value must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores.

    6. In the Toleration Seconds section, select one of the following options to specify how long a pod stays bound to a node that has a node condition.

      • Forever - Pods stays permanently bound to a node.

      • Custom value - Enter a value, in seconds, to define how long pods stay bound to a node that has a node condition.

    7. Click Add.

  8. If your accelerator profile contains existing tolerations, you can edit them.

    1. Click the action menu (⋮) on the row containing the toleration that you want to edit and select Edit from the list.

    2. Complete the applicable fields to update the details of the toleration.

    3. Click Update.

  9. Click Update accelerator profile.

Verification

  • If your accelerator profile has new identifying information, this information appears in the Accelerator list on the Start a notebook server page.

Additional resources

Deleting an accelerator profile

To discard accelerator profiles that you no longer require, you can delete them so that they do not appear on the dashboard.

Prerequisites

  • You have logged in to Open Data Hub as a user with Open Data Hub administrator privileges.

  • The accelerator profile that you want to delete exists in your deployment.

Procedure

  1. From the Open Data Hub dashboard, click Settings → Accelerator profiles.

    The Accelerator profiles page appears, displaying existing accelerator profiles.

  2. Click the action menu (⋮) beside the accelerator profile that you want to delete and click Delete.

    The Delete accelerator profile dialog opens.

  3. Enter the name of the accelerator profile in the text field to confirm that you intend to delete it.

  4. Click Delete.

Verification

  • The accelerator profile no longer appears on the Accelerator profiles page.

Additional resources

Viewing accelerator profiles

If you have defined accelerator profiles for Open Data Hub, you can view, enable, and disable them from the Accelerator profiles page.

Prerequisites

  • You have logged in to Open Data Hub as a user with Open Data Hub administrator privileges.

  • Your deployment contains existing accelerator profiles.

Procedure

  1. From the Open Data Hub dashboard, click Settings → Accelerator profiles.

    The Accelerator profiles page appears, displaying existing accelerator profiles.

  2. Inspect the list of accelerator profiles. To enable or disable an accelerator profile, on the row containing the accelerator profile, click the toggle in the Enable column.

Verification

  • The Accelerator profiles page appears appears, displaying existing accelerator profiles.

To help you indicate the most suitable accelerators to your data scientists, you can configure a recommended tag to appear on the dashboard.

Prerequisites

  • You have logged in to Open Data Hub as a user with Open Data Hub administrator privileges.

  • You have existing notebook images in your deployment.

  • You have enabled GPU support. This includes installing the Node Feature Discovery and NVIDIA GPU Operators. For more information, see NVIDIA GPU Operator on Red Hat OpenShift Container Platform in the NVIDIA documentation.

Procedure

  1. From the Open Data Hub dashboard, click Settings → Notebook images.

    The Notebook images page appears. Previously imported notebook images are displayed.

  2. Click the action menu (⋮) and select Edit from the list.

    The Update notebook image dialog opens.

  3. From the Accelerator identifier list, select an identifier to set its accelerator as recommended with the notebook image. If the notebook image contains only one accelerator identifier, the identifier name displays by default.

  4. Click Update.

    Note

    If you have already configured an accelerator identifier for a notebook image, you can specify a recommended accelerator for the notebook image by creating an associated accelerator profile. To do this, click Create profile on the row containing the notebook image and complete the relevant fields. If the notebook image does not contain an accelerator identifier, you must manually configure one before creating an associated accelerator profile.
Verification

  • When your data scientists select an accelerator with a specific notebook image, a tag appears next to the corresponding accelerator indicating its compatibility.

To help you indicate the most suitable accelerators to your data scientists, you can configure a recommended accelerator tag for your serving runtimes.

Prerequisites

  • You have logged in to Open Data Hub as a user with Open Data Hub administrator privileges.

  • You have enabled GPU support. This includes installing the Node Feature Discovery and NVIDIA GPU Operators. For more information, see NVIDIA GPU Operator on Red Hat OpenShift Container Platform in the NVIDIA documentation.

Procedure

  1. From the Open Data Hub dashboard, click Settings > Serving runtimes.

    The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled in your Open Data Hub deployment. By default, the OpenVINO Model Server runtime is pre-installed and enabled in Open Data Hub.

  2. Edit your custom runtime that you want to add the recommended accelerator tag to, click the action menu (⋮) and select Edit.

    A page with an embedded YAML editor opens.

    Note
    		 You cannot directly edit the OpenVINO Model Server runtime that is included in Open Data Hub by default. However, you can clone this runtime and edit the cloned version. You can then add the edited clone as a new, custom runtime. To do this, click the action menu beside the OpenVINO Model Server and select Duplicate.

  3. In the editor, enter the YAML code to apply the annotation opendatahub.io/recommended-accelerators. The excerpt in this example shows the annotation to set a recommended tag for an NVIDIA GPU accelerator:

    metadata:
	annotations:
		opendatahub.io/recommended-accelerators: '["nvidia.com/gpu"]'

  4. Click Update.

Verification

  • When your data scientists select an accelerator with a specific serving runtime, a tag appears next to the corresponding accelerator indicating its compatibility.