AI software development ROI: the 4 numbers that measure it

AI software development ROI comes down to four numbers that need no code reading, plus the contract clauses to demand from a vendor before you sign.

Review cost happens here, and it almost never shows up in the tool's ROI calculation

The ROI of AI in software development fits into a one-line division. On top, how many changes reached production this month and were still standing 30 days later, with no hotfix and no rollback. On the bottom, everything you paid to get them there: tool licences, the hours of whoever wrote the code, the hours of whoever reviewed it, the hours of whoever came back to fix it. Lines of code appear on neither side of that fraction.

To fill in the division you need four numbers, and none of them require reading code: the time between the first commit on a change and that change serving real users, the share of changes that come back within 30 days, the human time a reviewer spends per pull request, and the full cost per surviving change. Measure monthly and compare against the quarter before the tool landed on the team. If you never saved that quarter, start measuring today and accept that the first meaningful number arrives in about three months.

27 companies, plenty of spend, and the question showing up late

Nobody measures a build by concrete poured per day, they measure it by structure passing the load test

The Pragmatic Engineer published an analysis last week pulling data from 27 companies to answer exactly this: AI spending jumped at most companies this year, so did it move shipping velocity and PR output? What interests me there is not the finding, it is the order of events. The purchase came first. The ROI question came a year later, once the line item was already in everyone's budget.

The pattern holds outside the US. Surveys of executives in Brazil this month show the same shape: licences handed out to almost the whole engineering org, and almost no agreed definition of what would change if those licences were cancelled in March. Forty seats approved, zero sentences written about what success looks like.

I spent part of my career as the engineer of record on structural steel for a racetrack. Nobody measured that job by cubic metres of concrete poured per day. They measured it by structure passing the load test. Volume of material consumed is a warehouse metric, and software just spent two years treating volume of generated code as if it were an engineering result.

The four numbers that fit in one meeting

None of them require you to open a file. All four come out of tools your team already pays for, and the first three can be pulled from Git and the board in under a morning.

• End-to-end time per change: from the first commit on a branch to that change running for a real user. This is the number AI is supposed to crush, and the one that stubbornly refuses to move, because the bottleneck walked over to the review queue and left the keyboard alone. If you have no baseline, our PR review time benchmark for remote teams gives you an order of magnitude to compare against.
• 30-day return rate: out of every 100 changes that shipped, how many needed a hotfix, a revert, or a second pass through the same file for the same reason. This is the metric that exposes fake gains, because a change that ships fast and comes back next week got counted twice as productivity.
• Human hours per pull request: how long a reviewer actually spends per PR, and how many PRs they clear per day. A 900-line pull request approved in roughly four minutes is not a review, it is a signature. This number is what turns saved typing into added reading cost, and it is the one nobody tracks.
• Cost per surviving change: add the month's licences, the hours of whoever wrote the code, and the hours of whoever reviewed and repaired it, then divide by the changes still standing after 30 days. For the hourly part, use your real payroll cost, and if you are comparing against offshore or nearshore software development, our developer rates in Brazil gives you the market band.

The fourth number is the only one your CFO reads without translation, and the only one that closes the division from the top of this article. The other three exist so you know where it came from on the month it gets worse.

A green dashboard with no instrumentation behind it

The four numbers fit in a monthly meeting, and the fourth is the only one finance reads without translation

There is a pattern I find in almost every system I inherit after the previous vendor walks away: the health endpoint returns 200, the dashboard has six green tiles, and not one of them is instrumented. From the outside, full monitoring. Under the hood, nothing being observed.

AI productivity dashboards are heading the same way. Suggestion acceptance rate, lines generated, percentage of code written by an agent: all of it is the 200 on that health check. The metric exists, the guarantee does not. In January the slide showed how many lines the AI produced. By July the slide had quietly left the deck, because somebody asked what it had changed in the invoice or the delivery date and nobody had an answer.

And the bill those dashboards hide does not come due the month you buy. It comes due when someone has to change that code. If the previous vendor is the one who generated it, our 30-minute audit for software vendors is the fastest way to find out what you actually own.

"That's just DORA with extra steps"

Partly, yes. Two of the four numbers are cousins of delivery metrics that have been around for years, and I am not going to claim invention where there was none. Two things are different.

The first is the denominator. DORA measures the flow of delivery, not its cost. When the tool runs 30 to 40 dollars per seat per month and the org has 25 engineers, cost stops being a footnote and becomes half of the thing you are trying to decide.

The second is the 30-day survival window. Deployment frequency on its own rewards whoever ships fast, regardless of what shipped. Crossing shipped changes with 30-day returns is what separates output from outcome, and it is the same reason I keep pushing back on vendor reports that count throughput. The team they sold you is not always the team that ships, and I wrote about that specific gap here.

There is one cost none of these four calculations catch: your team understands less and less of the system it maintains. That shows up as longer bug investigations, as dependency on one or two people, as questions nobody in the channel can answer. I have paid for the same feature three times for reasons close to this one, and the money was the least of it.

What to put in the contract before you sign

If the people writing your software are a custom software development vendor rather than your own payroll, you cannot measure any of this alone. The data lives in their Git, their board, their credit card. So it goes into the agreement before signing, not during the first argument.

• Read access to the repository and the board, on your side, from day one. Without it you receive the four numbers pre-cooked in a PDF, and a number cooked by the party being graded on it is not a measurement.
• A per-change declaration of what was AI-generated. A label on the pull request is enough. This is not about banning anything, it is so you can compare 30-day returns between generated work and hand-written work.
• A written review policy, with a maximum PR size and two mandatory reviewers on anything touching payments, authentication, or personal data. If the answer is "the AI reviews it", ask for one week of approval logs and read them.
• Pass-through of the gain. If the tooling saves hours, those hours show up as a lower invoice or as extra scope. If they show up as neither, the gain belongs to the vendor and the risk still belongs to you.

That last item decides the rest. A contract model with no continuity clause rewards shipping fast and leaving, and AI only sped up the bad half of that. With a fixed price and no gain-sharing language, the incentive is to close the scope sooner, bill the same, and leave maintenance to whoever comes next.

Where this ruler breaks

On a three-person team shipping two or three pull requests a week, do not bother. The noise beats the signal, the 30-day return rate swings 30 points because one person took a holiday, and you end up making a 15 thousand dollar call on a sample of eight changes. The honest measurement there is qualitative: ask the team what got easier and what got more annoying, and decide on that.

It also fails in month one. Every new tool makes end-to-end time worse before it makes it better, because the team is still learning where it helps and where it lies. Give it a quarter before you look at the fraction.

And there is the case where the maths comes out negative and the right call is to keep paying. If the return does not land in delivery time but lands in who stays, who accepts the offer, who stops interviewing elsewhere, that is a return. It is just not an engineering return, and it should not be defended as one.

If you cannot mix concrete, it makes no difference whether you use a hoe or a mixer: the result is bad either way, and the mixer only gets you there faster. The question I would carry into the next budget meeting is the simplest of the four: how many of last month's changes are still standing today, with nobody having gone back to fix them? If nobody on the team can answer, cancelling the licences and renewing them are the same bet, placed in the dark.