Argo CD self-heal can roll out a release without its migration hook when a sync window closes
Maya Okonkwo
If your Argo CD apps run database migrations as Sync hooks, with self-heal and deny sync windows switched on, a release can reach the cluster without its migration and still show green. Afzal Ansari, a CNCF Ambassador, traced the failure in a post on the CNCF blog and reproduced it on demand in a lab. His fix is to stop depending on the hook.
Five regions, two skipped migrations
A customer pushed a release while a deny window was active. When the window opened, five regional Applications rolled out the same commit. Three ran the database migration. Two did not, and for 36 to 44 minutes the new code talked to a schema it did not understand.
The UI gave no warning. Both broken Applications were Synced and Healthy on the new commit, and their last operation read "successfully synced (all tasks run)". The only trace sat in the controller log. The two bad operations listed thirteen specific resources and carried a self-heal attempt count of one. Argo CD had healed drift instead of deploying a release, and a drift heal does not run hooks.
Ansari then rebuilt the setup in a lab and pushed a release during a four-minute window and refreshed the app fifteen seconds before it closed. The app synced in one second. The migration Job still carried the previous release label. He got the same result in five of six attempts.
Two refresh paths, one wrong assumption
Argo CD refreshes an Application for two reasons. A level-2 refresh comes from the reconciliation timer, roughly every two to three minutes, and compares the app against Git. A level-1 refresh is triggered by a change to a managed object and reuses what the controller already knows. After a window closes, whichever kind arrives first decides what happens.
While the window is shut, timer refreshes record the new commit in status.sync.revision, but the last real sync stays on the old commit. A level-1 refresh then compares that recorded revision against itself, finds no changes, and decides the revision was already synced. With self-heal on, it syncs only the out-of-sync resources. That partial sync skips hooks and writes no history. A timer refresh compares against targetRevision instead, sees that the last sync is older, and runs a full sync with hooks.
In Ansari's words: "On the level-1 path, the last compared revision is mistaken for the last synced one."
Why the obvious tweaks don't help
The quick fixes stay inside the hook model: a PreSync hook, the newer RunHooksOnPartialSync option, or a shorter window. Going by his reading of the Argo CD source, none of them changes the outcome. A partial sync skips every hook type. The new option is ignored for automated syncs, and self-heal is automated. He says plainly that he did not test these three.
Put the migration in Git
The recommended fix makes the migration part of the desired state: remove the hook annotations, give the Job a sync wave that runs before the app, and change its name every release:
apiVersion: batch/v1
kind: Job
metadata:
name: db-migration-<release> # must change every release
annotations:
argocd.argoproj.io/sync-wave: "-1"
Each release now brings a Job that does not exist yet, so any sync, partial or full, sees it as missing and applies it, and Argo CD waits for that wave to be healthy before moving on. In the lab, this version went through the same broken self-heal path and the migration still finished before the new ConfigMap and Deployment were applied.
The post lists the details to get right:
- Use a unique name per release, such as the image tag in a Helm template, because a Job's pod template is immutable.
- Keep prune on so old Jobs get cleaned up.
- Do not set
ttlSecondsAfterFinished. Kubernetes deletes the Job, self-heal sees a missing resource, and the migration runs again. - Keep migrations idempotent anyway.
- Run a deliberately failing migration in staging before relying on this.
The author did not test the failing case. That is where your rollback path lives, so test it yourself before the next release.
If you can't change the charts yet
The blunter option is to turn off self-heal on the apps that run migrations:
syncPolicy:
automated:
prune: true
selfHeal: false
Ansari ran two apps through the same window, and both made the same faulty "no changes" decision. The app with self-heal on shipped the release without its migration. The other logged "Skipping auto-sync: most recent sync already to […]" and waited. About two minutes later its timer refresh ran a full sync with the hook, and the migration finished. That app stayed on the old release while it waited, so code and schema never mismatched.
The cost is drift correction. A resource someone edits by hand now stays OutOfSync until the next commit.
The post does not point to an upstream fix. Until one exists, a Synced and Healthy status after a window close tells you that an operation succeeded. It does not tell you which kind of operation it was.
Source: CNCF Blog (cncf.io)