Kairos pipeline runs a Kubernetes control-plane upgrade in eleven minutes, no operator

An immutable-OS Kubernetes cluster upgraded its three control-plane nodes in eleven minutes with no operator in the loop, according to a CNCF case study published this week by Olivier Calzi. The wall-clock capture is small (one cluster, one bump), but the pipeline it describes is the interesting part. A two-line YAML change on a Kairos-flavoured cluster kicked off image verification, admission-policy checks and sequential node reboots without anyone SSH-ing in to watch etcd.

What actually ran

The nodes run Kairos, an immutable Linux distribution built to boot A/B partitions. On upgrade, the operator writes a new OS image to the inactive partition and reboots into it. Rollback is booting the old partition again. The upgrade itself is orchestrated by kairos-operator on the cluster, which drains and reboots each node in turn.

The trigger chain, per the post, is standard GitOps. Renovate watches upstream image tags and opens a pull request against the cluster's config repo, Gitea runs the CI, and ArgoCD reconciles the merged commit onto the cluster. Kyverno gates admission and Cosign verifies the image signature before the operator touches a node. The change that started the eleven-minute run was two lines of YAML.

The numbers the post reports

The wall-clock capture from the run:

• image pull: 25.5 seconds for a 324 MB OS image
• three control-plane nodes upgraded sequentially, concurrency of 1
• etcd quorum broken: never
• workload disruption: none
• total wall-clock time: eleven minutes

Concurrency of 1 is the load-bearing setting here. With three control-plane nodes and one reboot at a time, the cluster stays at 2/3 etcd members healthy, which is the minimum for quorum. Set the concurrency higher and this pattern falls over.

What the immutable-OS layer buys you

The traditional in-place upgrade path (apt-get upgrade, restart kubelet, hope) is what makes cluster upgrades a scheduled event with a runbook and a war-room bridge. A package pins a kernel your CSI driver dislikes. A systemd unit ordering changes. Someone left a manual edit in /etc that gets clobbered. Rolling back means restoring from backup or reprovisioning the node.

The A/B partition model collapses that. The new state is a signed image, the old state is still on disk, and rollback is a reboot. It is the same trick embedded devices and ChromeOS have used for years, applied to a cluster node. Combined with a GitOps trigger, the human step becomes reviewing a Renovate PR.

Where the pattern bites

Two caveats worth stating directly.

First, "zero humans" is measured post-merge. The PR still needs a reviewer, and the reviewer needs to know what bumping the OS image from X to Y means for the workloads on top. That review is where the operational judgment lives.

Second, the eleven-minute figure is for the control plane on one cluster. Worker-node fleets are where the wall-clock time grows and where PodDisruptionBudget-aware draining, storage attachment and workload eviction start to matter. The case study is a proof of pattern, not a benchmark for a hundred-node data plane.

Other projects hit the same problem from different angles. Talos Linux ships an immutable OS with its own upgrade API. Cluster API drives node replacement by rolling machine deployments through a controller. Bottlerocket uses A/B partitions on the worker side. The Kairos pipeline in the CNCF post is one specific composition. The general lesson is that when the OS is a signed artifact and the trigger is a git commit, a cluster upgrade becomes a code review.

Whether that lesson survives contact with your fleet depends on how boring your reviewers can keep the PR.