How we develop with AI agents — and the safeguards that turned out to be necessary

This text describes a pipeline that runs in production here — on a multi-tenant communication system built on ASP.NET Core, Angular and FreeSWITCH, with self-hosted GitLab. Every finding below comes from a real run and was found and fixed before the merge. We do not name the system, because individual findings would otherwise be traceable back to it.

What is the actual bottleneck in developing with AI? #

Not the speed at which code appears. The bottleneck is trust: code that compiles, whose tests are green and that reads correctly — and is wrong anyway. This class of defect accounts for almost every finding we have, and no standard pipeline sees it.

It has acquired a name here, because it recurs so reliably:

Something isn’t guarding, and nothing says so.

A pipeline is green. A guard does not hold. A checker checks something other than what its name claims. The pipeline described below is built against exactly that — not against slow typing.

How are the roles divided? #

Four roles with different models and different rights. The separation of rights is the load-bearing structure, not the choice of model.

Two of these separations are not cosmetic:

Implementers have no GitLab access. They commit into an isolated Worktree and report a hash. Push, Merge Request, labels, merge and deploy are done exclusively by the lead. That forces every change through exactly one bottleneck — and ensures nobody waves their own work through.

Reviewers may not change anything. They find and prove, they do not repair. A reviewer who repairs along the way stops looking. That is not a hunch; it is the same reason humans review their own code badly.

How does a ticket travel through the pipeline? #

Ticket (Definition of Ready)
  └─ Triage: feasible? → InProgress | not feasible? → remove DoR + comment on what is missing
      └─ Map the Seams (read-only agent)
          └─ Implementer in their own Worktree → Commit
              └─ Lead pushes → feature/<n> → draft MR against the integration branch
                  └─ Review PAIR over `git diff <base>...<tip>`
                      └─ Every finding as an inline discussion on the line + verdict as an MR note
                          └─ ONE prioritised fix batch to the SAME implementer
                              └─ ⟳ Round 2 over the FIX DIFF (not the whole MR)
                                  └─ Pipeline green → undraft → approve → merge
                                      └─ Full CI run on the integration branch → deploy UAT
                                          └─ Verification note on the ticket → label UAT

The point of this: the review lives on the Merge Request, not in a chat log. Every finding with a line reference becomes a resolvable GitLab discussion on exactly that line — with a concrete scenario, not with “this could be problematic”. Anyone opening that MR in six months sees finding → repair → resolution, just as with a human reviewer. For acceptance in environments with an obligation of evidence, that is the difference between a tool and a procedure.

Why is there another review after every fix? #

Because repairs have a defect class of their own. This is the most important rule in our pipeline, and the one that has found the most:

After every fix batch the review runs again — over the fix diff, not over the whole MR — with two different questions: what did the repair break? And are the regression tests real?

On a single working day this round produced the same pattern of finding three times: the repair closes one axis and tears open the neighbouring one.

• A guard was built into the offer gate; the invocation gate stayed open — even though the author had documented that very mistake two paragraphs earlier.
• A seed was closed on the host axis and open on the tenant axis, while the architecture documentation listed it as closed.
• Before one repair, foreign traffic could only make a check greener; afterwards an anonymous stranger could pin it to amber permanently.

After at most three rounds we stop iterating. Then a human decides. Iterations that still produce findings after the third round are a signal about how the ticket was cut, not about the code.

When does a regression test count as real? #

When it is proven to fail the moment you take the fix back out. “Test added” is not a statement. “Mutation X introduced, N tests red, mutation reverted, all green” is one. The proof belongs on the finding, not in the commit message.

That sounds like bureaucracy until you see the hit rate. Four examples from real runs:

• A checker for “the secret must not appear in the audit log” let the regression pass in 45 % of runs. Cause: the random Base64 key contains a + with roughly 49 % probability, and the JSON serializer escapes exactly that — so the substring check never matched. 22 runs to measure it, then a structural check: property present, yes or no.
• A concurrency test killed its mutant in 5 of 30 runs, because the time window never reopened after the first hit. Replaced with a barrier over 1000 rounds: 30 of 30.
• A checker looked for a returned foreign string only in public properties — the only realistic place for it to be stored would have been a private field.
• Five checkers belonging to a “hardening” commit all stayed green with a mutation in place. That commit existed because those same checkers had already measured too little once before.

A test that has never been red is not a safeguard. It is an assertion painted green.

Which classes of defect keep recurring? #

Every defect class that appears twice gets written down and handed to every review assignment. That list is the pipeline’s memory — agents have none. An excerpt:

• Registration by naming convention only takes effect when the class name matches. Otherwise the resolved list is empty, and the check happily reports “nothing registered”.
• “Not red” does not catch a dead checker. ShouldNotBe(Fail) also passes on Unknown. And typeof(X).ShouldNotBeAssignableTo<Y>() checks the type object — always green.
• Mutation probes against stale artefacts measure nothing. Twice in one day: once restored via mv (older mtime, MSBuild does not rebuild), once the mutation failed to compile and the “green runs” were no-ops.
• The twin has the guard, this instance does not. The single most expensive finding of one day was exactly that: a secret ended up in plain text in the audit log, while the matching attribute sat next to it in five sibling DTOs — one of them carrying the comment “must NOT be serialised into the audit log”.

Why is CI the gate and not the Worktree? #

Because a green partial run in a Worktree says nothing about the interplay. Merging happens on a green pipeline, and the integration branch then runs the full suite again, which also gates the UAT deploy.

Part of that is keeping the pipeline honest:

• Heavy jobs are serialised (resource_group). Four parallel feature pipelines stretched a 25-minute run to over an hour — two jobs died at exactly 3607 seconds in the runner cap. Four serial runs that complete are cheaper than four parallel ones that all fail.
• Superseded pipelines are cancelled (interruptible); jobs with external effects explicitly are not.
• A timeout is not a test failure. That distinction has to be made deliberately, otherwise you “repair” code that was never broken.

Why is state read back instead of assumed? #

Because a successful write call is not proof. The state you read back is. Ticket labels (DoR → InProgress → UAT) are set here by a small script that changes both sides in one call and reads the state afterwards.

That script came into being after the lead had broken the rule by hand three times. It then had two defects of its own: an abort exactly during verification, and a check that, because of pagination, only looked at the first 100 tickets and still reported “all in order”. The same defect class the tool was meant to prevent, inside the tool.

We consider that the most honest evidence that this defect class has nothing to do with AI. Speed only makes it more frequent.

Which decisions stay with humans? #

Architecture, security and release — plus every blocking question. Blocking questions go out as a ticket comment, with two or three options, a recommendation and a statement of what happens without an answer. Not as a chat message that disappears into the log.

Hard rules no agent softens:

• Production data is read, not written. Configuration changes are named by the agent as a GUI field; setting them is done by hand.
• Credentials live only in a file outside the repository and are read inline — never in a commit, MR, comment or log.
• No pushes to protected branches. No deploy without a green pipeline.

What did the pipeline actually find? #

A selection from one run, all before the merge:

One pattern is so reliable that it now serves as a search heuristic: if deleting a guard outright breaks the build, the dangerous mutation is not the omission but the swapped condition.

What did not work? #

Honesty belongs here, otherwise this becomes advertising.

• One-shot alarms on individual pipelines. Every new push creates a new ID that nobody is waiting on. Replaced by a persistent watch on the project.
• Overload instead of parallelism. More concurrent agents does not mean more throughput when they all push against the same machine. One agent shot down other people’s test runs while cleaning up.
• The lead is not immune. Several wrong instructions in fix batches were refuted by the implementers — with measurements. One of them would have introduced a regression, another would have turned a checker from reliable into random. That implementers may contradict and must prove it is part of the safeguard.
• Diagnoses asserting one of several possible causes cost hours. “Tunnel down” was the message; the cause was a missing route in the reverse proxy. Error messages now state status code and content type instead of a guess.

The short version #

1. Separation of rights: whoever builds does not push. Whoever reviews does not change.
2. Two reviewers, two models, two perspectives — on the same diff.
3. Every finding becomes a resolvable discussion on the line, not a chat message.
4. After every fix, the fix is reviewed, with different questions than the first time.
5. A test counts when it demonstrably fails the moment you take the fix back out.
6. Recurring defect classes are written down and handed to every assignment.
7. State is read back. A successful write call is not proof.
8. After three rounds a human decides, not the next iteration.

The effort is real: a feature easily goes through three review rounds, and every round costs. The return is real too — these are consistently findings a green pipeline does not deliver.

That is also the answer to the question clients ask us most often about AI. The capacity we free up goes into test depth, not into more features. How we introduce this into existing teams and toolchains is described under AI engineering; the framework for regulated projects under The guardrails we set for agents and Quality assurance & test automation.