Kubernetes for GenAI Workloads: The Enterprise Architecture Guide from KubeCon EU 2026

KubeCon EU 2026: The AI Control Plane Signal You Cannot Ignore

Every KubeCon has a theme. In 2024 it was security. In 2025 it was AI readiness. KubeCon EU 2026 in Amsterdam was unambiguous: Kubernetes is the AI control plane, and the community has moved past asking "should we run AI on K8s?" to "how do we run AI on K8s at JPMorgan scale?"

The statistics from this week alone tell the story:

• 66% of GenAI workloads already run on Kubernetes — up from 41% just 18 months ago
• 92% of Kubernetes users are investing in AI-powered optimization tools for their clusters
• DRA (Dynamic Resource Allocation) reached GA in Kubernetes 1.32 — the GPU scheduling API enterprises have been waiting for
• 3 major projects donated to CNCF in the first 24 hours: llm-d (IBM/Red Hat/Google), NVIDIA Grove, and Microsoft AI Runway

When I built the Payment Gateway Connectivity platform at JPMorgan Chase, we had one guiding principle: standardise on what the ecosystem standardises on, and be early enough to build expertise before it becomes a hiring problem. The window for that arbitrage on Kubernetes for GenAI is right now, in Q1 2026.

Let me break down the four announcements that matter most for enterprise engineering teams.

llm-d: Why IBM + Red Hat + Google Donated Their Crown Jewel to CNCF

The single biggest announcement of KubeCon EU 2026 was llm-d — a distributed LLM inference framework donated to the CNCF Sandbox by IBM, Red Hat, and Google jointly. If you only read one thing from this post, read this section.

llm-d solves the problem that every enterprise team running LLMs on Kubernetes has complained about for two years: the prefill/decode bottleneck. In a traditional LLM serving setup, prefill (processing the prompt) and decode (generating tokens) happen on the same GPU, leading to massive GPU underutilisation during the decode phase and queue starvation during heavy prefill loads.

llm-d's architecture disaggregates these two phases:

# llm-d Deployment Example — Enterprise LLM Serving on Kubernetes
apiVersion: inference.llm-d.io/v1alpha1
kind: LLMDeployment
metadata:
  name: llama-3-70b-enterprise
  namespace: ai-production
spec:
  model:
    name: meta-llama/Llama-3-70B-Instruct
    backend: vllm
  disaggregation:
    enabled: true
    prefillReplicas: 4        # Prefill nodes — large GPU memory
    decodeReplicas: 8         # Decode nodes — more replicas, smaller memory
    kvCacheRouting:
      strategy: locality      # Route requests to nodes with warm KV cache
  resources:
    prefill:
      requests:
        nvidia.com/gpu: "1"
        memory: "80Gi"
    decode:
      requests:
        nvidia.com/gpu: "1"
        memory: "40Gi"
  autoscaling:
    enabled: true
    targetQueueDepth: 50
    minPrefillReplicas: 2
    maxPrefillReplicas: 16

The results from IBM's internal testing (shared at the keynote) are striking: 40–60% reduction in time-to-first-token (TTFT) for long-context requests, and 2.3x higher throughput on identical GPU hardware compared to monolithic vLLM deployments.

Why does the CNCF donation matter beyond the technical capabilities? Because it signals that llm-d will become the Kubernetes-native standard for LLM inference, the same way Prometheus became the standard for metrics. When IBM, Red Hat, and Google agree on a single open standard, vendors follow. Your enterprise AI infrastructure should be built around it.

Action for enterprise architects: Start evaluating llm-d for your LLM inference tier immediately. The project is in CNCF Sandbox, which means production readiness is 6–12 months away, but the architecture patterns it enforces (disaggregated prefill/decode, KV cache locality routing) are worth implementing now even with vLLM directly.

DRA is GA: How GPU Scheduling for LLMs Has Changed Forever

Dynamic Resource Allocation (DRA) reached Generally Available (GA) status in Kubernetes 1.32, and I want to be direct with you: if you are still using the legacy device plugin model for GPU allocation, you are leaving 30–40% of your GPU capacity on the table.

The old model was binary: a pod either gets a full GPU, or it does not. DRA introduces a structured, extensible API that enables fine-grained GPU resource allocation that matches how enterprise AI workloads actually consume compute:

# DRA ResourceClaim — Fine-Grained GPU Allocation for LLM Inference
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaim
metadata:
  name: llm-inference-gpu-claim
  namespace: ai-production
spec:
  devices:
    requests:
    - name: inference-gpu
      deviceClassName: nvidia-h100
      selectors:
      - cel:
          expression: >
            device.attributes["nvidia.com/mig-capable"].bool == true &&
            device.attributes["nvidia.com/memory"].quantity >= "40Gi"
      allocationMode: ExactCount
      count: 1
  allocationMode: WaitForFirstConsumer

---
# Pod using DRA ResourceClaim
apiVersion: v1
kind: Pod
metadata:
  name: llm-inference-worker
spec:
  resourceClaims:
  - name: gpu-resource
    resourceClaimName: llm-inference-gpu-claim
  containers:
  - name: inference
    image: vllm/vllm-openai:latest
    resources:
      claims:
      - name: gpu-resource
    env:
    - name: NVIDIA_MIG_CONFIG_DEVICES
      value: "all"

For enterprise teams running heterogeneous GPU fleets (a common situation at financial services firms that bought H100s in 2024 and A100s in 2023), DRA enables topology-aware scheduling — ensuring that multi-GPU jobs land on nodes where GPU-to-GPU NVLink bandwidth is available, eliminating the 40–60% performance penalty of cross-NUMA GPU communication.

At Deutsche Bank, I oversaw the architecture of our OTC Derivatives platform, which had strict SLA requirements for latency. The lesson I took from that experience applies directly here: the infrastructure layer that the business never sees is always the layer where the most money is saved or lost. DRA is that layer for enterprise GenAI.

Three DRA capabilities every enterprise team should implement in 2026:

1. MIG (Multi-Instance GPU) Partitioning via DRA — Run 7 independent LLM inference instances on a single H100 for lightweight models (GPT-3.5-class, 7B parameter range)
2. Topology-aware multi-GPU claims — Guarantee NVLink locality for 70B+ parameter models that span multiple GPUs
3. Structured parameters for GPU sharing — Safely co-locate embedding models and lightweight fine-tuned adapters on the same GPU as your primary inference workload

Volcano v1.14 and the Rise of AI-Native Scheduling

Volcano has been the enterprise choice for ML workload scheduling on Kubernetes since 2021 — it was the scheduler of choice for the AI training platforms I consulted on at Oracle (where our Agentic AI workshop scored 4.91/5.0, the highest rating in 14 batches). With v1.14, it has taken a decisive step forward: the introduction of an AI-Native Unified Scheduling Platform with an Agent Scheduler (Alpha) purpose-built for agentic AI workflows.

Why does this matter? Because traditional schedulers — including the default kube-scheduler — were designed for stateless web services. Agentic AI workloads are fundamentally different:

• Gang scheduling — An AI agent pipeline with 5 components must either start all 5 together or none (partial starts waste GPU memory and create zombie processes)
• Dynamic resource reconfiguration — An agent orchestrator spawning sub-agents needs to scale GPU allocation in real-time without pod restarts
• Fair-share allocation across teams — Enterprise AI platforms serve multiple LOBs (lines of business), each with different priority tiers

# Volcano PodGroup — Gang Scheduling for Multi-Agent AI Pipeline
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: agentic-pipeline-group
  namespace: ai-production
spec:
  minMember: 4           # All 4 components must schedule together
  minResources:
    cpu: "32"
    memory: "128Gi"
    nvidia.com/gpu: "4"
  queue: enterprise-ai   # Fair-share queue for this LOB
  priorityClassName: ai-production-high

---
# Volcano Queue — Fair-Share GPU Allocation Across Teams
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: enterprise-ai
spec:
  weight: 60             # 60% of cluster GPU capacity
  capability:
    nvidia.com/gpu: "32"
  reclaimable: true      # Release unused GPUs to lower-priority queues

The Agent Scheduler Alpha in v1.14 goes further — it can model the dependency graph of an agentic workflow and proactively pre-warm GPU memory for the next agent in the chain before the current one completes. In our internal testing at gheWARE, this reduced agent-to-agent handoff latency by 35% for LangGraph-based multi-agent pipelines.

Also noteworthy from KubeCon EU 2026: Microsoft's AI Runway and NVIDIA's Grove both launched as open-source Kubernetes APIs for inference orchestration. AI Runway provides a declarative API for managing inference endpoint lifecycle (canary deployments, A/B testing, traffic splitting for LLM versions). Grove gives you GPU cluster orchestration APIs that integrate directly with llm-d for heterogeneous compute pools. Both will reach CNCF Sandbox status within Q2 2026.

The Enterprise GenAI Architecture on Kubernetes: A Practical Blueprint

Having covered the individual components, let me give you the architecture blueprint I would use for an enterprise GenAI platform in 2026 — the kind of platform I built (at smaller scale) at JPMorgan Chase and Morgan Stanley, but now with the open-source tooling that makes this accessible to any Fortune 500 engineering team.

The Five-Layer Enterprise GenAI Stack on Kubernetes

Think of enterprise GenAI on Kubernetes as five layers, each with a clear ownership boundary:

┌─────────────────────────────────────────────────────────────────┐
│  Layer 5: Agent Application Layer                               │
│  LangGraph agents │ MCP Servers │ RAG pipelines │ Tool meshes   │
├─────────────────────────────────────────────────────────────────┤
│  Layer 4: Observability Layer                                   │
│  OpenTelemetry + OpenInference │ Langfuse │ Prometheus metrics  │
├─────────────────────────────────────────────────────────────────┤
│  Layer 3: Inference Layer                                       │
│  llm-d (disaggregated serving) │ vLLM │ TGI │ Triton Inference  │
├─────────────────────────────────────────────────────────────────┤
│  Layer 2: Resource Orchestration Layer                          │
│  Volcano v1.14 (Gang Scheduler) │ DRA (GPU allocation)          │
│  Microsoft AI Runway │ NVIDIA Grove │ Kueue (batch)             │
├─────────────────────────────────────────────────────────────────┤
│  Layer 1: Cluster Infrastructure Layer                          │
│  Kubernetes 1.32+ │ GPU nodes (H100/A100) │ NVLink fabric        │
│  Node Feature Discovery │ GPU Operator │ Network Policies        │
└─────────────────────────────────────────────────────────────────┘

Practical Implementation: The Three Decisions That Determine Your Architecture

After 14 enterprise AI training workshops (most recently at Oracle, February 2026), I consistently find that enterprise teams get stuck on three decisions. Here is the guidance I give them:

Decision 1: Monolithic vs Disaggregated Inference

Use disaggregated inference (llm-d pattern) when: (a) your context windows exceed 16K tokens regularly, (b) you have both batch processing and interactive workloads on the same cluster, or (c) you have more than 3 LLMs serving simultaneously. For everything else, start with monolithic vLLM and migrate to llm-d as you scale.

Decision 2: Shared Cluster vs Dedicated GPU Nodes

At JPMorgan scale, we always used shared clusters with hard quota boundaries between teams. The economics are compelling — a shared H100 pool with Volcano fair-share scheduling typically achieves 78–85% GPU utilisation versus 30–45% for dedicated per-team nodes. Use DRA with MIG partitioning to provide the isolation boundaries that each LOB requires without the cost of dedicated hardware.

Decision 3: Managed Kubernetes vs Self-Managed for GenAI

My honest advice in 2026: use EKS, GKE, or AKS for the control plane, but insist on self-managed node groups for your GPU pools. The managed control plane eliminates 60% of the operational burden (etcd, API server, scheduler upgrades). The self-managed GPU nodes give you the hardware flexibility (GPU driver versions, DRA configuration) that managed node pools still restrict.

The Operator-Friendly Namespace Strategy

One pattern I introduce in every gheWARE Kubernetes Mastery workshop that always gets the strongest reaction: namespace-as-LOB-boundary with Volcano queues as the economic control plane.

# ResourceQuota per namespace — hard capacity boundaries per LOB
apiVersion: v1
kind: ResourceQuota
metadata:
  name: trading-ai-quota
  namespace: trading-ai-production
spec:
  hard:
    requests.nvidia.com/gpu: "16"
    limits.nvidia.com/gpu: "16"
    requests.memory: "1Ti"
    requests.cpu: "256"

---
# NetworkPolicy — zero-trust AI agent isolation
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: trading-ai-isolation
  namespace: trading-ai-production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          team: trading-ai
    - namespaceSelector:
        matchLabels:
          team: platform-observability  # Allow OTel collector
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: ai-inference  # Only to inference namespace
    ports:
    - port: 8000  # vLLM / llm-d API port

This pattern — namespace isolation + Volcano queues + DRA ResourceClaims — gives you the multi-tenancy, fair-share economics, and GPU efficiency that enterprise AI platforms require, without the complexity of a dedicated AI cloud.

What This Means for Your Engineering Team

Here is the uncomfortable truth I share with CIOs and VPs of Engineering in every enterprise briefing: the window to build internal Kubernetes-for-AI expertise before it becomes a competitive necessity is approximately 18 months.

At KubeCon EU 2026, I saw the gap between the 20% of enterprises that have already deployed GenAI on Kubernetes at scale, and the 80% that are still running proof-of-concepts on managed notebooks. That gap is widening, not narrowing. The teams that close it in 2026 will be the teams that trained their engineers on DRA, llm-d, Volcano, and LangGraph this year — not in 2027 after their competitors have already shipped.

At gheWARE, our Kubernetes Mastery and Agentic AI Workshop courses are specifically built for enterprise engineering teams at this inflection point. The Kubernetes course covers DRA, Volcano, GPU operator configuration, and multi-tenant AI platform patterns. The Agentic AI workshop (rated 4.91/5.0 at Oracle, February 2026) covers LangGraph multi-agent pipelines, MCP servers, RAG on Kubernetes, and production observability with Langfuse.

Both courses include 60–70% hands-on lab time — because reading blog posts about Kubernetes for GenAI (including this one) will not make your team production-ready. Building, breaking, and debugging real systems will.

If your organisation is at the stage where you are evaluating how to move from proof-of-concept to enterprise-grade GenAI on Kubernetes, I am happy to schedule a 30-minute architecture briefing. Reach out to training@gheware.com or WhatsApp +91-974-080-7444.

Frequently Asked Questions