The Scheduler Has Never Heard of the Fabric: Topology-Aware GPU Placement for AI Inference

Your LLM serving cluster runs 40% slower than the same replica configuration measured inside a single fabric domain. There is no Kubernetes alert for this, no dashboard signal, no failing health check. The usual response is to provision more GPUs.

Kubernetes treats compute as fungible. A node with eight GPUs in rack A is scheduled identically to one in rack B, blind to whether inter-GPU communication crosses a high-speed fabric domain, a congested PCIe switch, or a RoCE-over-Ethernet path shared with everyone else's traffic. For multi-tenant inference - tensor-parallel serving replicas, disaggregated prefill/decode - that blindness decides whether P99 latency meets SLA or collapses under peak load.

This talk builds topology-aware scheduling for inference serving. We expose physical topology from fabric management APIs and switch LLDP data, configure Kueue ResourceFlavors and gang placement so replicas land inside one fabric domain, and confront the problem nobody mentions: what happens after weeks of churn, when no clean domain is left. We cover RoCE fabrics and heterogeneous hardware, not just pristine InfiniBand, and show how to derive real Model FLOPs Utilization from DCGM counters so this degradation stops being invisible.