GitLab shipped a real migration with AI agents and wrote up what actually broke

The last time I watched a team migrate a piece of production plumbing across a hundred call sites, we spent most of the first week just naming everything before anyone typed a real line of code. So when GitLab posted this week about running the same kind of exercise with AI agents doing most of the mechanical fan-out, I was ready to be either annoyed or hooked. The post lands on the honest side of the fence, and that is the reason it is worth reading.

The setup, per GitLab's write-up: two separate rate-limiting paths in production. One is the application-level Gitlab::ApplicationRateLimiter with 121 keys. The other is a Rack-level system further out on the edge, which the post says handles roughly four billion requests a day. Both were being unified onto a single implementation in the labkit-ruby gem, and that meant touching 95 call sites across the monolith and the EE codebase. It is the kind of migration where a missed key does not fail loudly. It just quietly stops working the way you thought.

What the agents were actually asked to do

The agents were not turned loose on the codebase. GitLab describes a bounded set of moves they were trusted with, driven from the Duo Agent Platform and paired with Duo Code Review and Duo Developer:

• Read context and draft specifications from the epic.
• Implement bounded code changes, one spec at a time.
• Write test code alongside the change.
• Pre-review the merge requests before a human ever looked at them.
• Fan out mechanical edits across the 95 call sites (83 in the monolith, 12 in EE).

Nothing in that list is "the agent decided the plan". The plan is still an engineering artifact. What moves is the fan-out and the first-draft review, which happen to be the two parts of a migration that eat the most calendar time and cause the most typos at line 4,203.

The safety loop they wrapped around it

The interesting bit is the process, not the model. The post describes a structured loop that reads more like a runbook than a prompt. Read the epic, write a spec, adversarially review the spec, implement, verify with explicit evidence in the merge request, adversarially review the MR, human review, merge. Adversarial passes are capped at two rounds of resolution before a human takes over, so an agent cannot argue with itself forever.

That cap is the sentence I circled twice. If you have ever watched two review bots trade suggestions back and forth in a comment thread, you know why it matters.

The rollout they actually ran

Nothing about this migration was one-shot. The team split the keys into cohorts and rolled them out at 1%, then 10%, then 50%, then 100%, hitting the halfway point on May 4 and the full cutover on May 5, per the post. There were six cohorts in total, because a first audit missed 17 keys and Cohort 6 became the sweep for the ones nobody had noticed.

The relevant numbers for anyone thinking about this at home, in GitLab's own count: 14 numbered specs, and north of 30 merge requests. This was not one heroic prompt with a green pipeline. It was a long series of small, reviewable steps.

The incident they did not hide

The post reports one shadow-mode failure. An adapter quietly dropped an identifier on one unauthenticated code path, and a slice of users started seeing generic failures before the team rolled it back. GitLab names it. That single admission is more useful than a pile of eval numbers, because it points at the exact class of thing an AI agent is bad at catching: the seam between two systems where a value looks harmless in isolation and turns dangerous when it flows across the boundary.

Shadow mode caught it before it hurt anyone at scale, which is the whole point of shadow mode. Still, if you read the write-up expecting a clean line that says "look, no incidents", you will not find one, and I would trust the post less if you did.

What lands for the engineer waiting on the queue

There is nothing in this migration that a careful team could not do by hand. The agents did not invent the plan. They did not choose the cohorts. They did not decide when to flip percentages. What they took off the humans was the mechanical fan-out across 95 call sites and the first pass of spec writing, inside a loop that a human still owned end to end.

If you have ever run a monotonous migration for a week, you know exactly which minutes those are. They are the ones where you lose your place and the compiler cannot save you.

The reason this post reads well to me is that it does not claim the agents replaced engineering judgment. It claims they compressed the fan-out and the first-draft review. That is a much more useful claim, and it is one you can copy without buying anyone's roadmap.

What I am watching next

Two things. First, whether the "adversarial review, capped at two rounds" pattern shows up in write-ups from other teams. It is a small idea with a lot of leverage, and I want to see it stress-tested against messier codebases than a Ruby monolith with strong review culture. Second, whether the seam-of-two-systems failure mode ends up as a named test category in agent evaluation, the way race conditions eventually got their own bucket in async test suites.

If your next agent-assisted change touches a boundary between two subsystems, borrow this team's rollout shape before you borrow their tooling. The percentages and the audit cohort are the parts that would have saved them either way.