Customize models to build gen AI applications

Red Hat AI includes a maintained Python package index that provides secure and reliable access to supported libraries, with full support for disconnected environments. For details about Red Hat support for the Python package, see Support philosophy: A secure platform.

Table 2.1 lists the images that are configured to use the Red Hat Python index.

Notes:

If you are using a disconnected environment, use the following code example to access the Red Hat Python index content and copy it locally. You can then upload the packages into your own internal hosting service:

To ensure reliable and secure access to supported libraries, start your model customization workflow by creating a workbench image that is based on a Red Hat base image that is configured to use the Red Hat Python index. These base images are listed in Table 2.1.

Note: When you create a custom workbench image that is based on one of the images listed in Table 2.1, install JupyterLab. You can use JupyterLab to run the example model customization notebooks.

For guidance on custom workbenches, see Creating a custom workbench image from your own image.

When you use one of the images listed in Table 2.1 as a base image, both pip and uv commands are pre-configured to use the Red Hat Python index and system trust store for HTTPS.

When you run a pip install command, it installs the package version referenced in the Red Hat Python index, ensuring that you are installing a version of the library that is secure and reliably accessible.

For example, use the following commands to install the model customization libraries:

To get started with customizing your models, you can run provided example notebooks and scripts. The following table lists the Git repositories that provide example notebooks for each model customization component.

For a comprehensive tutorial that demonstrates an AI/ML workflow, see the Knowledge Tuning example on the Red Hat AI examples site.

The Knowledge Tuning tutorial is a curated collection of Jupyter notebooks that includes examples of using Docling to process data, Training Hub to fine-tune a model on that data, and KServe to deploy the final model for a Question and Answer application.

Docling is the Python library that you use to prepare unstructured data (like PDFs and images) for consumption by large language models.

Additional resources

With Kubeflow Pipelines (KFP), you can automate complex, multi-step Docling data processing tasks into scalable workflows.

With the KFP Software Development Kit (SDK), you can define custom components and stitch them together into a complete pipeline. The SDK allows you to fully control and automate Docling conversion tasks with specific parameters.

Note: You can build a custom runtime image to ensure that all required Docling dependencies are present for pipeline execution. For information on how to run a Docling pipeline with a custom image see the Docling Pipeline documentation.

NOTE: If you want to use a Red Hat container image in one of the above pipelines, replace the image on this line with the URL for the Red Hat Docling container image and recompile the pipeline that you want to use.

To get started with SDG Hub, explore the provided examples.

Additional resources

To get an estimate of the total time a flow will take, you can run the dry_run function and set enable_time_estimation to true.

For example, tests that use the gpt-oss-120b LLM on 4x H100 GPUs with the QuALITY dataset (266 articles) showed significant variance between flows.

To view a graph that illustrates the accuracy on QuALITY Benchmark (4,609 Evaluation QA), go to: https://github.com/Red-Hat-AI-Innovation-Team/sdg_hub/blob/main/examples/knowledge_tuning/enhanced_summary_knowledge_tuning/imgs/quality_benchmark_accuracy.png.

You can generate synthetic data for domain-specific model customization by using a Kubeflow Pipeline (KFP) on Open Data Hub. The Domain Customization Data Generation using Kubeflow Pipelines (KFP) is a guided example.

The example pipeline generates three types of document augmentations and four types of QA on top of 3 augmentation and 1 original document. It stores the generated data in the Cloud Object Storage (COS) bucket that is linked through the pipeline server.

The Training Hub repository hosts multiple cookbooks for using different LLM algorithms such as Supervised Fine-tuning (SFT), Orthogonal Subspace Fine-tuning (OSFT)/Continual Learning, and Low-Rank Adaptation (LoRA)/Quantized Low-Rank Adaptation (QLoRA). OSFT is a training algorithm built by the Red Hat AI Innovation team. With OSFT, you can continually post-train a fine-tuned model to expand its knowledge on new data. You can tinker with the Training Hub cookbooks from a workbench within your Open Data Hub project.

To get started with Training Hub, explore the provided examples.

To simplify tuning for enterprise customers, Training Hub supports multiple backends and exposes a unified API surface to access the latest training algorithms from different backends.

The following table lists Training Hub algorithm and model support matrix.

NOTE: If you experience an issue with the model classes listed for OSFT with use_liger=True, try setting use_liger=False. Liger kernels are supported for most model architectures, but some newer architectures might experience errors or instability if not fully supported. For up-to-date support information, see the Liger-Kernel GitHub repository.

To learn how to estimate the amount of memory you need for running and training a specific model, as well as whether your configured GPUs can support the model, use the memory estimator. The memory estimator currently supports only Supervised Fine-tuning (SFT) and Orthogonal Subspace Fine-tuning (OSFT) algorithms. See the following example files in the Training Hub Git repository:

You can use the Orthogonal Subspace Fine-Tuning (OSFT), Supervised Fine-Tuning (SFT), and Low-Rank Adaptation (LoRA) algorithms in Training Hub.

Use SFT to fine-tune a model on supervised datasets with support for:

Use OSFT to fine-tune a model while controlling how much of its existing behavior to preserve, with support for:

Use LoRA for parameter-efficient fine-tuning with significantly reduced memory requirements, with support for:

The examples/docs directory contains information and examples for how to use each algorithm.

Here is a performance comparison of using OSFT, SFT, and LoRA in Training Hub.

NOTE: When scaling the usage of Liger Kernels for all methods, some amount of fixed overhead memory is added to all methods that do not use Liger Kernels.

If you want to implement distributed training across multiple nodes to meet the needs of your training workloads, you can use the Kubeflow Training Operator (KFTO). KFTO abstracts the underlying infrastructure complexity of distributed training and fine-tuning of models. The iterative process of fine-tuning significantly reduces the time and resources required compared to training models from scratch.

Learn more about KFTO in the following Open Data Hub documentation:

The Kubeflow Training Operator (KFTO) supports distributed fine-tuning by using Training Hub, abstracting the complexity of distributed training. It seamlessly manages scaling and orchestration for you, allowing you to focus on your domain-specific fine-tuning logic by using the simplified Training Hub APIs.

For a comprehensive tutorial on Fine Tuning with Training Hub leveraging distributed nodes with KFTO, follow these guided examples: