Working with distributed workloads

In a terminal window, create a directory for your work, and change to that directory.

Set the IMG environment variable to the name of your image. In the example commands in this section, my_training_image is the name of the image.

export IMG=my_training_image

Create a file named Dockerfile with the following content:

Build the image file. Use the -t option with the podman build command to create an image tag that specifies the image name and version, to make it easier to reference and manage the image:

podman build -t <image-name>:_<version>_ -f Dockerfile

podman build -t ${IMG}:0.0.1 -f Dockerfile

Display a list of your images:

podman images

Push the image to your container registry:

podman push ${IMG}:0.0.1

Optional: Make your new image available to other users, as described in Pushing an image to the integrated OpenShift image registry.

In a terminal window, log in to the OpenShift CLI as shown in the following example:

$ oc login <openshift_cluster_url> -u <admin_username> -p <password>

Set the IMG environment variable to the name of your image. In the example commands in this section, my_training_image is the name of the image.

export IMG=my_training_image

Log in to the integrated image registry:

podman login -u $(oc whoami) -p $(oc whoami -t) $(oc registry info)

Tag the image for the integrated image registry:

podman tag ${IMG} $(oc registry info)/$(oc project -q)/${IMG}

Push the image to the integrated image registry:

podman push $(oc registry info)/$(oc project -q)/${IMG}

Retrieve the image repository location for the tag that you want:

oc get is ${IMG} -o jsonpath='{.status.tags[?(@.tag=="<TAG>")].items[0].dockerImageReference}'

In the JupyterLab interface, click File > New > Notebook, and then click Select.

Add the following code to a cell in the new notebook:

from codeflare_sdk import copy_demo_nbs
copy_demo_nbs()

Select the cell, and click Run > Run selected cell.

In the left navigation pane, right-click the new notebook and click Delete.

Click Delete to confirm.

In the left navigation pane, double-click demo-notebooks.

Double-click additional-demos and verify that the folder contains several demo notebooks.

Click demo-notebooks.

Double-click guided-demos and verify that the folder contains several demo notebooks.

Check whether your cluster administrator has defined a default local queue for the Ray cluster:

In the JupyterLab interface, open the demo-notebooks > guided-demos folder.

Open all of the notebooks by double-clicking each notebook file.

In each notebook, ensure that the import section imports the required components from the CodeFlare SDK, as follows:

from codeflare_sdk import Cluster, ClusterConfiguration, TokenAuthentication

In each notebook, update the TokenAuthentication section to provide the token and server details to authenticate to the OpenShift cluster by using the CodeFlare SDK.

Optional: If you want to use custom certificates, update the TokenAuthentication section to add the ca_cert_path parameter to specify the location of the custom certificates, as shown in the following example:

auth = TokenAuthentication(
    token = "XXXXX",
    server = "XXXXX",
    skip_tls=False,
    ca_cert_path="/path/to/cert"
)
auth.login()

In each notebook, update the cluster configuration section as follows:

In the 2_basic_interactive.ipynb notebook, ensure that the following Ray cluster authentication code is included after the Ray cluster creation section.

from codeflare_sdk import generate_cert
generate_cert.generate_tls_cert(cluster.config.name, cluster.config.namespace)
generate_cert.export_env(cluster.config.name, cluster.config.namespace)

Run the notebooks in the order indicated by the file-name prefix (0_, 1_, and so on).

The notebooks run to completion without errors.

In the notebooks, the output from the cluster.status() function or cluster.details() function indicates that the Ray cluster is Active.

Create a data connection to connect the object storage to your data science project, as described in Adding a data connection to your data science project.

Configure a pipeline server to use the data connection, as described in Configuring a pipeline server.

Create the data science pipeline as follows:

Import your data science pipeline, as described in Importing a data science pipeline.

Schedule the pipeline run, as described in Scheduling a pipeline run.

When the pipeline run is complete, confirm that it is included in the list of triggered pipeline runs, as described in Viewing the details of a pipeline run.

In the Open Data Hub left navigation pane, click Distributed Workloads Metrics.

From the Project list, select the project that contains the distributed workloads that you want to monitor.

Click the Project metrics tab.

Optional: From the Refresh interval list, select a value to specify how frequently the graphs on the metrics page are refreshed to show the latest data.

In the Requested resources section, review the CPU and Memory graphs to identify the resources requested by distributed workloads as follows:

Scroll down to the Top resource-consuming distributed workloads section to review the following graphs:

Scroll down to view the Distributed workload resource metrics table, which lists all of the distributed workloads in the selected project, and indicates the current resource usage and the status of each distributed workload.

In the Open Data Hub left navigation pane, click Distributed Workloads Metrics.

From the Project list, select the project that contains the distributed workloads that you want to monitor.

Click the Distributed workload status tab.

Optional: From the Refresh interval list, select a value to specify how frequently the graphs on the metrics page are refreshed to show the latest data.

In the Status overview section, review a summary of the status of all distributed workloads in the selected project.

Scroll down to view the Distributed workloads table, which lists all of the distributed workloads in the selected project. The table provides the priority, status, creation date, and latest message for each distributed workload.

In the OpenShift Container Platform console, in the Administrator perspective, click Observe → Alerting.

Click the Alerting rules tab to view a list of alerting rules for default and user-defined projects.

Click the name of an alerting rule to see more details, such as the condition that triggers the alert. The following table summarizes the alerting rules for Kueue resources.

If the Alert state of an alerting rule is set to Firing, complete the following steps:

In a terminal window, if you are not already logged in to your OpenShift cluster, log in to the OpenShift CLI as shown in the following example:

$ oc login <openshift_cluster_url> -u <username> -p <password>

Configure a training job, as follows:

Create the training data.

Create a persistent volume claim (PVC), as follows:

In the OpenShift Container Platform console, select your project from the Project list.

Click ConfigMaps and verify that the training-config and twitter-complaints ConfigMaps are listed.

Click Search. From the Resources list, select PersistentVolumeClaim and verify that the trained-model PVC is listed.

In a terminal window, log in to the OpenShift CLI as shown in the following example:

$ oc login <openshift_cluster_url> -u <username> -p <password>

Create a PyTorch training job, as follows:

In the OpenShift Container Platform console, select your project from the Project list.

Click Workloads → Pods and verify that the <training-job-name>-master-0 pod is listed.

In the OpenShift Container Platform console, select your project from the Project list.

Click Workloads → Pods.

Search for the pod that corresponds to the PyTorch job, that is, <training-job-name>-master-0.

Click the pod name to open the pod details page.

Click the Logs tab to monitor the progress of the job and view status updates, as shown in the following example output:

0%| | 0/10 [00:00<?, ?it/s] 10%|█ | 1/10 [01:10<10:32, 70.32s/it] {'loss': 6.9531, 'grad_norm': 1104.0, 'learning_rate': 9e-06, 'epoch': 1.0}
10%|█ | 1/10 [01:10<10:32, 70.32s/it] 20%|██ | 2/10 [01:40<06:13, 46.71s/it] 30%|███ | 3/10 [02:26<05:25, 46.55s/it] {'loss': 2.4609, 'grad_norm': 736.0, 'learning_rate': 7e-06, 'epoch': 2.0}
30%|███ | 3/10 [02:26<05:25, 46.55s/it] 40%|████ | 4/10 [03:23<05:02, 50.48s/it] 50%|█████ | 5/10 [03:41<03:13, 38.66s/it] {'loss': 1.7617, 'grad_norm': 328.0, 'learning_rate': 5e-06, 'epoch': 3.0}
50%|█████ | 5/10 [03:41<03:13, 38.66s/it] 60%|██████ | 6/10 [04:54<03:22, 50.58s/it] {'loss': 3.1797, 'grad_norm': 1016.0, 'learning_rate': 4.000000000000001e-06, 'epoch': 4.0}
60%|██████ | 6/10 [04:54<03:22, 50.58s/it] 70%|███████ | 7/10 [06:03<02:49, 56.59s/it] {'loss': 2.9297, 'grad_norm': 984.0, 'learning_rate': 3e-06, 'epoch': 5.0}
70%|███████ | 7/10 [06:03<02:49, 56.59s/it] 80%|████████ | 8/10 [06:38<01:39, 49.57s/it] 90%|█████████ | 9/10 [07:22<00:48, 48.03s/it] {'loss': 1.4219, 'grad_norm': 684.0, 'learning_rate': 1.0000000000000002e-06, 'epoch': 6.0}
90%|█████████ | 9/10 [07:22<00:48, 48.03s/it]100%|██████████| 10/10 [08:25<00:00, 52.53s/it] {'loss': 1.9609, 'grad_norm': 648.0, 'learning_rate': 0.0, 'epoch': 6.67}
100%|██████████| 10/10 [08:25<00:00, 52.53s/it] {'train_runtime': 508.0444, 'train_samples_per_second': 0.197, 'train_steps_per_second': 0.02, 'train_loss': 2.63125, 'epoch': 6.67}
100%|██████████| 10/10 [08:28<00:00, 52.53s/it]100%|██████████| 10/10 [08:28<00:00, 50.80s/it]

The <training-job-name>-master-0 pod is running.

The Logs tab provides information about the job progress and job status.

In the OpenShift Container Platform console, select your project from the Project list.

Check the workload resource:

Check the Ray cluster resource:

Check the cluster queue resource:

In the OpenShift Container Platform console, select your project from the Project list.

Click Search, and from the Resources list, select Pod.

Click your pod name to open the pod details page.

Click the Events tab, and review the pod events to identify the cause of the problem.

If you cannot resolve the problem, contact your administrator to request assistance.

In the OpenShift Container Platform console, select your project from the Project list.

Check the workload resource:

Check the Ray cluster resource:

In the OpenShift Container Platform console, select your project from the Project list.

Click Search, and from the Resources list, select LocalQueue.

Resolve the problem in one of the following ways:

In the OpenShift Container Platform console, select your project from the Project list.

Click Search, and from the Resources list, select LocalQueue.

Resolve the problem in one of the following ways:

Identify the correct OpenShift login credentials as follows:

In your notebook code, specify the copied token and server values as follows:

auth = TokenAuthentication(
    token = "<token>",
    server = "<server>",
    skip_tls=False
)
auth.login()

In the OpenShift Container Platform console, select your project from the Project list.

Click Search, and from the Resources list, select Pod.

Click the Ray head pod name to open the pod details page.

Click the Events tab, and review the pod events to check whether the image pull completed successfully.