Distributed Inference with llm-d

Red Hat AI Inference 3.5

Architecture, components, and deployment of Distributed Inference with llm-d for scalable LLM serving on Kubernetes

Abstract

Learn about Distributed Inference with llm-d, a Kubernetes-native framework for serving large language models at scale.

Preface

Distributed Inference with llm-d is a Kubernetes-native framework for serving large language models at scale. You can deploy Distributed Inference with llm-d on Openshift Container Platform or managed Kubernetes platforms such as Azure Kubernetes Service (AKS) and CoreWeave Kubernetes Service.

Important

Distributed Inference with llm-d is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Chapter 1. Enterprise-grade inference serving

The Distributed Inference with llm-d framework provides enterprise-grade large language model (LLM) inference serving on Openshift Container Platform and managed Kubernetes clusters on public clouds such as Azure Kubernetes Service (AKS) and CoreWeave Kubernetes Service (CKS).

Enterprise platform engineering and infrastructure teams can use Distributed Inference with llm-d to build generative AI model services for internal and external use cases. Cloud service providers can also use it to build Models-as-a-Service (MaaS) offerings. Common use cases include:

Enterprise-wide Models-as-a-Service (MaaS) for generative AI: A central platform team provides generative AI and LLM capabilities as a managed service to business units across the organization. Rather than each team provisioning its own inference infrastructure, the platform team uses Distributed Inference with llm-d to offer standardized model serving with consistent performance, cost control, and security.
Production-ready inference at scale: An organization deploys a generative AI application in a limited production environment, such as A/B testing with a small user group or a soft launch. The deployment must be production-ready with reliable performance and security, while remaining flexible enough to scale to provider-grade inference as demand grows.

Chapter 2. Inference scheduling and caching capabilities

Distributed Inference with llm-d provides intelligent scheduling, caching, and resource management capabilities for distributed inference. These features optimize GPU usage, reduce inference latency, and enable cost-effective scaling of large language models.

Intelligent inference scheduling: Provides prefix-cache aware routing that directs each request to the replica most likely to have relevant KV cache entries already populated, maximizing GPU KV cache reuse. The inference scheduler evaluates GPU utilization metrics, queue depth, cache residency, service level agreement (SLA) constraints, and load distribution across nodes to select the optimal replica for each request.
KV cache management: Manages key-value cache efficiently across distributed inference servers, reducing memory requirements and enabling longer context windows. Routing requests to replicas with warm KV cache entries avoids redundant prompt processing, which improves both throughput and time-to-first-token.
Prefill-decode disaggregation: Separates the compute-intensive prefill phase from the latency-sensitive decode phase, allowing you to assign each phase to appropriately optimized resources and scale them independently. The prefill phase processes the full input prompt in parallel and is assigned to compute-optimized resources. The decode phase generates tokens incrementally and is assigned to latency-optimized resources. This phase-aware architecture increases GPU utilization, reduces tail latency, and lowers cost per token.
Important
Prefill-decode disaggregation is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Wide expert parallelism: Supports efficient distributed inference of mixture of experts (MoE) models across many GPU nodes, enabling cost-effective scaling of large models.
Important
Wide expert parallelism is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Chapter 3. Distributed Inference with llm-d core components

Distributed Inference with llm-d deploys a distributed inference stack through Helm charts packaged as OCI container images, integrating industry-standard Kubernetes components to provide intelligent model serving, secure communication, and scalable resource management.

The Distributed Inference with llm-d inference stack consists of the following core components:

Red Hat AI Inference: High-throughput, low-latency LLM inference engine that runs in worker pods and processes model inference requests.
KServe LLMISvc Controller: Kubernetes-native model serving control plane that defines the LLMInferenceService custom resource and manages model lifecycle, deployment orchestration, and operational governance. KServe reconciles the required serving state declared in the custom resource, coordinates component deployment, and exposes OpenAI-compatible APIs. Request routing decisions are handled by the inference scheduler, also known as the Endpoint Picker (EPP).
Inference scheduler (Endpoint Picker): Scheduling service that makes per-request routing decisions by integrating with the gateway through the ext-proc protocol. The scheduler uses a pluggable system of filters, scorers, and profile handlers to evaluate KV cache locality, queue depth, session affinity, and load distribution when selecting optimal model-serving pods.
Routing sidecar: Sidecar service deployed alongside each model-serving pod that orchestrates disaggregated inference execution. The routing sidecar coordinates prefill and decode stages, manages KV cache transfers between pods, and provides endpoint protection.
Gateway API with Red Hat Connectivity Link: Kubernetes standard API for traffic routing and load balancing, with the Gateway API Inference Extension specification for intelligent request routing. Red Hat Connectivity Link provides the Gateway API implementation.
Istio/Sail Operator: Service mesh that provides mutual TLS (mTLS) encryption, traffic routing, and observability across the inference stack.
cert-manager: Automates Transport Layer Security (TLS) certificate management and provides a Certificate Authority for mutual TLS (mTLS) communication between inference components.
LeaderWorkerSet: Kubernetes workload API for coordinating distributed inference servers and managing pod topology for prefill-decode disaggregation and expert parallelism.
Note
The LeaderWorkerSet Operator is only required for wide expert parallelism (wideEP) with Distributed Inference with llm-d. With LeaderWorkerSet, you can do multi-node deployments across multiple AI accelerator nodes.

Chapter 4. Distributed inference workloads

Distributed Inference with llm-d uses a layered architecture to distribute inference workloads across a Kubernetes cluster.

Client requests flow through an inference gateway that routes the requests to optimized model replicas, with intelligent scheduling and autoscaling to manage resources efficiently. The gateway provides OpenAI-compatible APIs with streaming token responses. Body-based routing selects the InferencePool from the model name specified in the API request.

The inference pool contains model-serving variants, such as Prefill and Decode, each running Red Hat AI Inference instances. Variants handle different phases of the inference process. Shared prefix caching enables key-value (KV) cache sharing between model-serving variants.

Figure 4.1. Distributed Inference with llm-d processing architecture

Distributed Inference with llm-d system architecture

Body-based routing selects the InferencePool from the model name in the OpenAI-compatible API spec.

The inference scheduler is an extensible framework of filters, scorers, and profile handlers that provides load, KV, and prefill/decode-aware routing. The scheduler evaluates GPU utilization, queue depth, cache residency, and service level agreement (SLA) constraints to direct each request to the optimal replica. Routing favors replicas with warm KV cache entries for the request’s prefix, maximizing cache reuse and avoiding redundant computation.

Each decode pod asynchronously reports KV cache availability and load metrics to the inference scheduler.

Chapter 5. Separation of control plane and data plane

The Distributed Inference with llm-d architecture separates the model serving control plane from the inference data plane.

KServe manages model lifecycle, scaling, and API exposure. The llm-d inference scheduler handles runtime-aware scheduling, cache locality optimization, and intelligent request distribution across pods and nodes. This separation enables platform teams to swap runtimes or schedulers independently and integrate future innovations without redesigning the stack.

You deploy the inference stack by using Helm charts distributed as OCI container images, making it compatible with Openshift Container Platform 4.19 or later and any Cloud Native Computing Foundation (CNCF) certified managed Kubernetes 1.33 or later cluster. On Openshift Container Platform, the chart integrates with Operator Lifecycle Manager (OLM) to install required Operators automatically. On managed Kubernetes, the chart installs all dependencies directly.

Chapter 6. Distributed Inference with llm-d deployment with Helm

The Distributed Inference with llm-d Helm chart deploys a complete inference stack on Openshift Container Platform or managed Kubernetes. On Openshift Container Platform, the chart uses Operator Lifecycle Manager (OLM) to install and configure required Operators automatically. On managed Kubernetes, the chart installs all dependencies directly.

The Helm chart uses a three-tier deployment model:

Red Hat OpenShift AI Operator: The chart installs the Red Hat OpenShift AI Operator through an OLM Subscription. The chart controls the DataScienceCluster and DSCInitialization custom resources that manage the inference stack lifecycle.
Helm chart components: The Helm chart includes the rhaii profile, which provides an inference-focused deployment. For distributed inference, the primary component is KServe, which provides the LLMInferenceService custom resource (CR) for deploying and managing inference services.
Operator dependencies: Each component declares the Operators it requires. KServe depends on cert-manager, LeaderWorkerSet, and Red Hat Connectivity Link. The Helm chart resolves these dependencies, including transitive dependencies, and installs each Operator through OLM automatically.

Figure 6.1. Deploying Distributed Inference with llm-d with Helm charts

Helm chart deployment architecture showing the RHAI Operator and RHAI cloud controller manager components

The Red Hat AI Inference (RHAII) Helm chart packages, deploys, and configures the Red Hat AI (RHAI) Operator and the RHAI cloud controller manager. The RHAI Operator handles KServe and model serving, while the RHAI cloud controller manager handles the underlying cluster infrastructure.

The RHAI Operator manages the KServe Controller and reconciles custom resource definitions (CRDs).

The RHAI cloud controller manager manages Helm-based infrastructure components, including cert-manager, Gateway API, Istio, and LeaderWorkerSet.

The RHAI cloud controller manager configures the managed Kubernetes or Openshift Container Platform cluster.

Legal Notice

Except as otherwise noted below, the text of and illustrations in this documentation are licensed by Red Hat under the Creative Commons Attribution–Share Alike 3.0 Unported license . If you distribute this document or an adaptation of it, you must provide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, the Red Hat logo, JBoss, Hibernate, and RHCE are trademarks or registered trademarks of Red Hat, LLC. or its subsidiaries in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

XFS is a trademark or registered trademark of Hewlett Packard Enterprise Development LP or its subsidiaries in the United States and other countries.

The OpenStack® Word Mark and OpenStack logo are trademarks or registered trademarks of the Linux Foundation, used under license.

All other trademarks are the property of their respective owners.