#kubeflow

The 60% idle GPU that turned out to be a network policy

A CNCF write-up walks through a distributed Kubeflow training job whose GPUs sat around 60% idle while every pod reported healthy. The failure lived one layer down, in a Cilium policy enforcing zone boundaries that gradient sync had to cross.