Six weeks ago I proposed a metric. The argument was that engineering organizations can measure AI spend to the cent and cannot measure what it produced, so cost quietly stands in for value. The fix I offered was to change the unit: cost per verified outcome, token spend divided by the changes that shipped and stayed shipped through an N-day survival window.
Then a hackathon gave me a reason to actually compute the denominator against real repositories, and the framework taught me something it had no way to tell me while it was still an essay.
You cannot observe survival. Nothing in your toolchain fires an event when a change is fine. You are not measuring survival at all. You are measuring the absence of failure evidence, and that is a fundamentally different thing with a fundamentally different failure mode.
The denominator is an inference, not a reading
Go back to the definition. A verified outcome is a merged change that spent N days in production without a revert, a hotfix, or a reopened ticket. Read it as an algorithm rather than a definition and the shape becomes obvious: you enumerate merged changes, you look for evidence of repudiation in a window, and everything you did not find evidence against gets filed as verified.
The verified column is the default. It is where changes land when your detection came up empty, and “came up empty” covers two completely different worlds. It covers the change that was genuinely fine. It also covers the change that broke something, got quietly repaired, and left no artifact your query could match.
Both land in the same column. The metric cannot distinguish them, and by construction it never will.
The errors on either side of that line do not get treated equally, and the asymmetry is worth being precise about, because my first instinct was to say something wrong here. Detection can absolutely fire when it should not. Later in this post there is a case where it tried to accuse sixteen changes at once and every one of them was innocent. So false failures are real.
But those are the errors you go hunting for. A false accusation attached to somebody’s work is the expensive kind, so every guard you build is aimed at suppressing them. Nobody builds a guard against a miss, and nobody can: the defining property of a miss is that you never saw it. False positives get caught, false negatives accumulate in silence, and the score drifts upward by roughly the size of your detection gap.
Which nobody measures. So the number is a ceiling: the best case consistent with the evidence you managed to collect.
A one-sided alarm
Once you accept the asymmetry, the honest thing is to stop treating the metric as a score and start treating it as an alarm that only rings in one direction.
A low verified share is informative. Every counted failure carries evidence you can open and read: this change, that revert, this reference, that timestamp. You can disagree with a specific verdict, and disagreement is the point, because the evidence is right there to argue with.
A high verified share is not informative in the same way. It is the sound of nothing being detected, which is compatible with excellent engineering and equally compatible with an instrument pointed at a wall.
That asymmetry should change how the number gets used. It is a tripwire, not a grade. You run it after a model change, a prompt change, or an autonomy change, on the same project with the same configuration, and you watch whether it moves. Within-project, within-cohort, fixed-configuration trend. The moment somebody puts two teams’ verified shares side by side on a slide, the metric has been misused, because you are comparing two detection gaps of unknown and probably different sizes.
The flip side is a warranty worth having. When the alarm does ring, it has already survived review. That matters more than it sounds, because the situation this metric is really built for is the one where a single bad incident is about to set the autonomy dial for two quarters. In that room, the useful artifact is not a score. It is a ledger: we looked this hard, with these channels open, and found only these failures, and here they are.
So publish how blind you are
If the score is a ceiling, then the number that makes it interpretable is the one nobody ships: how much failure could this instrument have seen?
The cheapest usable proxy is hygiene you already have. What share of merged changes link to a work item at all, and what share link to a bug or incident specifically? If a codebase rarely connects its fixes to its issues, then path-based and link-based failure detection has almost nothing to grip, and a perfect survival rate is the expected output regardless of how the code is doing.
Here is what that looks like on real data. On a live audit over 128 merged merge requests with a 7-day window, the verified share came out at 97.7%: 125 verified, 3 failed. A great number. Then the detectability section: 56 of 128 merged changes closed some work item (43.8%), and 3 of 128 closed a bug or incident work item (2.3%).
Two point three percent. The entire 97.7% rested on a bug-linking rate low enough that the report flagged it as thin failure evidence and told you not to read the headline as health. The instrument priced down its own demo, on the run I intended to show people.
I have come to think that is the feature and not the embarrassment. A survival metric without a detectability number beside it is an unfalsifiable claim. Adding the second number converts “we scored 97.7%” into the far more useful “we scored 97.7% while able to see almost nothing,” which is an operational finding: go link your fixes to your issues, then measure again and the number will mean something.
It also closes an exploit, though I should be precise about how wide it actually is. Revert detection reads commit lineage and does not care whether you file issues at all, so that channel is hard to starve. The channels that catch the quieter failures are the ones leaning on links between fixes and tickets, and those are cheap to starve indeed: link fewer fixes, detect fewer failures, watch your survival rate improve while nothing about the code changes. Publishing detectability makes that visible the moment it starts, whether somebody is doing it on purpose or your issue hygiene is just decaying the way everyone’s does.
The failure that nothing sees
Reverts are the easy case and the wrong thing to design around. A revert leaves clean lineage: a commit that references another commit, a description that says what it undoes, timestamps on both ends. Detecting it is close to free.
The common failure looks nothing like that. An agent merges a change. Four days later a senior engineer notices it is subtly wrong and quietly repairs it in the course of other work. No revert. No rollback. No incident. No reopened ticket, because no ticket was ever opened. The repair may not even be recognized as a repair by the person doing it.
Every standard instrument is blind to this. Revert detection finds nothing because nothing was reverted. Change-failure rate finds nothing because no failure was declared. Adoption dashboards correlate tool usage with aggregate DORA-style rates and report that some percentage of changes failed, which does not tell you which change, on what evidence, or in which cohort. The most common way agent-written code goes wrong is also the way that leaves no trace.
The only channel I found that even attempts it is diff path overlap: when a later change is linked to a bug or incident, look at which files it touched, and check whether an earlier merged change inside the window touched the same distinctive files. If they overlap, the earlier change is implicated.
Implicated, not convicted. This is a correlational signal and it must be labeled as one everywhere it appears. Two changes touching the same file is evidence of proximity, not causation, and the correct output is a claim with its full chain attached so a human can adjudicate it. On that same run it produced 4 counted implications across 3 changes, each printing which files overlapped and which fix implicated which original.
A metric that counts failures has to defend its own precision
The first time the overlap logic ran on live data it tried to report sixteen failures at once.
One merge request closed a documentation-accuracy bug issue and touched a broad set of files, so it overlapped sixteen earlier merged changes. Sixteen failures, from one signal, in one shot. The verified share would have collapsed and the entire drop would have been an artifact.
What it actually was: a release merge doing a batch close. The guard that catches this is unglamorous. Cap the number of originals a single fix may implicate, and when a fix exceeds the cap, demote the whole cascade to advisory instead of counting any of it, on the reasoning that a fix implicating sixteen things is far more likely to be a batch-close signature than sixteen simultaneous defects. Sixteen would-be failures went to the ledger as demoted, uncounted, and visible.
The generalizable lesson is about which direction to be paranoid in. A metric that counts failures is making an accusation every time it increments, and false accusations are expensive in a way false clean bills are not. One bogus failure attached to somebody’s change destroys trust in the whole instrument, and you do not get it back by explaining the heuristic. So you tune for precision over recall, you accept that the score is an upper bound, and you say so loudly. Every counted failure prints a chain you can click. Everything the guards catch stays on the ledger as demoted, because a suppressed signal that nobody can see is its own kind of dishonesty.
I used to think of precision over recall as a tuning decision. It is closer to a social contract. The number has to survive inside an organization where people’s work is attached to it, and one false accusation is enough to end it.
The cohort split is the decision this serves
All of this exists for one question, and it is not a reporting question. As agents write more of the codebase, every team repeatedly decides whether to grant them more autonomy or less, and that decision currently runs on anecdotes and whichever incident is most recent.
So split every metric by who wrote the change: autonomous agent, human using an AI assistant, or human. Same window, same evidence rules, three columns.
On that 128-change population the split was 26 AI-assisted at 92.3% verified against 102 human at 99.0%.
Now the discipline, because this is exactly the number that escapes into a slide deck. That entire spread rests on three counted failure events across the whole population. Two failures wide. It is one project, merge-verified rather than deploy-verified because the namespace had no production deployment timestamps. And it is observational, confounded by task mix in a way that belongs in the headline rather than the footnotes. Teams assign agents the work they already trust agents with, so the two cohorts were never doing comparable work in the first place.
Stratifying by change size localizes both AI-assisted failures to the bucket above 500 lines, where the human cohort also does worse. Large changes fail across cohorts. That is a much duller finding than the headline gap and considerably more likely to be true.
The right way to read a gap that thin is as a prompt to go open the three failures and adjudicate them, not as a verdict about AI code quality. Which is why the tool now prints the basis count and flags any cross-cohort gap resting on fewer than five events as directional. A metric that cannot restrain its own interpretation will be over-interpreted by someone with a quarterly target.
What this does to CPVO
The essay proposed cost per verified outcome with one companion metric. Verified share sat next to it so that a high-volume team could not hide failed work behind a large numerator of survivors. CPVO answers what a verified outcome costs; verified share answers whether most changes survive.
Building it convinced me that pair is incomplete. There should be a third number on the dashboard, and it should sit at the end of that same sentence: CPVO tells you what a verified outcome costs, verified share tells you whether most changes survive, and detectability tells you how much failure either number was in a position to see.
Drop the third and the first two decay quietly. As issue hygiene slips, the link-dependent detection channels see less, the denominator grows, and cost per verified outcome improves on a dashboard while nothing changes about the engineering. No adversary required. Ordinary entropy produces the same result, and the metric would read it as progress.
There is a second correction. The survival window needs an anchor declared honestly. Ideally you anchor at deploy time, because production exposure is what the window is supposed to measure. Most repositories will not have that data, so you anchor at merge time and label the result merge-verified rather than deploy-verified. The temptation to quietly present a merge-anchored number as a production-survival number is strong, and it is the single easiest way to make this whole family of metrics dishonest. Declare which rung of the ladder you are standing on, per cohort, in the output.
Where it landed
The implementation is Stayed Shipped, built for the GitLab Transcend Hackathon on Orbit, GitLab’s knowledge graph. Survival accounting needs merge requests, deployments, work items, users, and pipelines joined in one place, which is otherwise five API integrations and a database, and is the reason the outcome side of the ledger was computable at all.
It took runner-up in the Technological Implementation category of the Showcase Track, announced on 13 July 2026. The judges described it as asking “a question your dashboards cannot answer: of the changes your AI agents merged last month, how many are still in production?”
That question is still the whole thing. What I did not appreciate when I wrote the essay is how much of the work is in the second half of the answer: not just producing a number, but publishing how much the number could not see. The reference engine for the framework is at github.com/sneg55/cpvo, and Stayed Shipped is the version that survived contact with real repositories, which mostly meant learning where it was blind and printing that beside every result.
Merged is not done. Shipped and stayed shipped is done. And knowing how hard you looked is part of knowing either one.
