The 60% idle GPU that turned out to be a network policy

CNCF today published a debugging narrative on a distributed Kubeflow training job whose GPUs sat around 60% idle while every pod insisted it was healthy, and the operational read for any platform team running training on Kubernetes is that CNI choices can pin your utilization below the floor of what pod-level dashboards will ever tell you. No crash. No OOMKill. Just wasted silicon.

The symptom the standard dashboards did not name

The setup, per the CNCF post, is a Kubeflow training job that shards gradient synchronization across GPU workers coordinated by a training coordinator. Under the hood, gradient sync uses NCCL AllReduce, the collective every pytorch-shaped training workload eventually falls back on. AllReduce is chatty. Every step, every GPU has to reach every peer.

That is where the network layer stops being background. The write-up notes that when the coordinator and the workers land in different Kubernetes zones, a CiliumNetworkPolicy enforcing a zone boundary can either deny the peer traffic outright or, in the milder version, force the collective across cross-zone links. The post says throughput drops 30 to 60 percent when the traffic goes cross-zone, and cross-AZ egress charges show up on the cloud bill as the third receipt.

Why node health had nothing to say

None of that shows up in the primitives node health teaches you to trust. The pods are Running. There is no restart count. Prometheus and Grafana, both referenced in the post, will happily draw a green line across the incident if the query is pod uptime. The signal is on the GPU: the collective sits waiting for a peer while the accelerator idles, and the wall-clock slows down without any layer above the network noticing.

At fleet scale, this is the failure mode that eats budget quietly. GPU capacity is scarce in a way general-purpose nodes are not, and every minute a training run spends waiting on the CNI is a minute a paid-for accelerator is drawing power for nothing.

The fix the CNCF post describes

The debug ends by co-locating the coordinator and the workers on the same GPU zone. The write-up names four Kubernetes primitives that do the work: nodeAffinity to pin the group to the GPU zone, topologySpreadConstraints to co-locate coordinator with workers, a toleration so the coordinator can land on GPU-tainted nodes, and podAffinity with a zone topology key as a portable alternative to hard-pinning. Per the post, GPU utilization moves from around 40 percent to around 85 percent after co-location.

The pipeline read

Two things fall out for anyone who owns a training platform. First, the CNI is part of the training workload, whether the training team owns it or not. A CiliumNetworkPolicy written to model a security boundary can become a performance gate the second gradient sync crosses it. If your CI is pushing a training pipeline through GitOps, the policy needs to live under the same review and pre-prod exercise as the training YAML itself, on the same kind cluster the post uses.

Second, the observability shape has to reach into the GPU. Pod-level metrics will not surface a stalled collective. NCCL telemetry and GPU-side exporters will, and folding them into the Prometheus scrape the platform team already runs is the cheapest addition that turns a silent incident loud.

The write-up links a repository and a conference talk recording with the full walkthrough. Cheaper to read once than to debug live on a schedule of expensive accelerators.