Where the controller-runtime cache bites you: stale reads, missing indexes, unbounded watches

A revised kubernetes.io deep dive on the controller-runtime cache landed on July 29, and the operational read is unchanged from the first version. Every Get and List your reconciler runs is answered by a local, eventually consistent copy of the cluster, not by the API server. That is why your controller does not knock over kube-apiserver at scale, and it is where a class of quiet reliability bugs actually lives.

What the cache actually is

The cache is a local in-memory view of Kubernetes objects, kept current by informers. An informer combines a reflector (one initial list plus a persistent watch) with an indexer that supports fast lookups by key or field. The manager wires it up per controller and populates it before the reconciler runs. Reads through r.Get() and r.List() inside Reconcile() hit that cache. Writes through r.Update() and r.Create() go straight to the API server. The authors, Andrei Kvapil and Timofei Larkin of Ænix, frame the cache as the reason a cluster with hundreds of custom controllers does not collapse under a thundering herd of direct API server queries.

Reads look free, and mostly are

Get is a keyed lookup against the local indexer. Cost is close to zero, and it is fine to call it repeatedly per reconcile. List is where the assumption breaks. Without a matching index, the runtime falls back to a linear scan over the cached objects. On a cluster with tens of thousands of Pods or CRs that is measured in CPU time, not microseconds, and every reconciler sharing the cache pays that cost on every triggered event. The article's guidance is direct: if you list with field selectors, define the index.

The write-then-read trap

The cache updates asynchronously through the watch stream. After r.Update() succeeds against the API server, a follow-up r.Get() can return the object as it was before the write. The window is small (the article puts it at milliseconds to seconds), but the reconciler has to be designed for it. Two shapes of bug follow. First, a reconciler that patches a Status field and immediately re-reads it to compute the next transition. Second, a reconciler that treats "no matching object" from the cache as authoritative and creates a duplicate. The escape hatch is the APIReader, which bypasses the cache and reads through to the API server. Use it sparingly. Every APIReader call is a request the cache was there to prevent.

Watches are the memory dial

Cache size scales with what you watch. Watch too many kinds or namespaces and the process grows into gigabytes of resident memory, silently, without a specific event to trigger a page. The article's recommendation is to be explicit about the object types and namespaces the manager caches, and to keep the index set tight. This is the knob CI/CD teams running Argo Workflows, Tekton, or Argo CD in the same cluster as tenant workloads reach for first. An operator that watches every Pod cluster-wide behaves very differently from one scoped to its own namespace, and the difference does not show up until steady-state memory is measured under load.

Eventual consistency is the contract

The last takeaway from the revised text is a design constraint, not a bug. The cache lags. Reconcilers have to be idempotent and tolerant of a stale read. The corollary for anyone shipping controllers into a shared cluster: reads are cheap only when the indexes exist, writes never round-trip through the cache, and the watch surface you configure at Manager time is the memory bound you carry in production.