← All posts

Explainer

Measuring whether a coding agent is actually helping

A month into using a coding agent, everyone on the team agrees it is faster. Nobody can say by how much, and the numbers that get quoted in the retro — lines shipped, suggestions accepted, hours saved — are the three worst available. That is not a coincidence. They are the three easiest to collect, and each of them rewards precisely the behaviour you would want a good agent to avoid.

There is one randomised controlled trial in this area worth reading, and its result is that experienced developers were 19% slower with AI while believing they had been 20% faster. That gap — not the slowdown, the gap — is the thing that should change how you measure. If your instrument is a person's recollection at the end of a sprint, it can be off by 39 percentage points in the flattering direction while everyone involved is honest and paying attention.

The three metrics that reward the wrong behaviour

Start with the ones your tooling offers you for free, because they are the ones you will otherwise use by default.

Lines written

An agent optimising for lines has an easy job. Inline the helper instead of extracting it. Duplicate the validation block rather than importing the one three files over. Write the exhaustive table-driven test file instead of the four cases that actually discriminate. Add the defensive null check the type system already guarantees. Every one of those is more lines, all of them plausible in review, and all of them make the codebase worse in the specific way that is hardest to see in a diff.

Meanwhile the best change an agent can make is frequently a deletion, which scores negative.

Two pieces of evidence, from opposite directions. GitClear's 2025 report analysed 211 million changed lines of code authored between January 2020 and December 2024, across a mix of private repositories and large open-source projects. It found that 2024 was the first year on record in which within-commit copy/paste exceeded moved lines, and recorded an eight-fold increase during 2024 in the frequency of commits containing a code block with five or more duplicated lines, with the prevalence of duplicate blocks roughly 10x what it had been two years earlier. Read that honestly: GitClear sells code-quality analytics, its dataset is not a random sample of software, and it cannot attribute any individual commit to an AI. It is a time series that correlates with industry-wide adoption, not an experiment. It is still the largest such time series anyone has published, and it points the same way as the mechanism above.

The other direction is sharper because it comes from a randomised trial. In METR's study, developers wrote 47% more lines of code per forecasted hour on the AI-allowed issues — and those were the issues that took longer. The paper notes the difference is not statistically significant (p = 0.16), so do not lean on the size of it. Lean on the sign. The arm that produced more code was the slower arm.

Acceptance rate

Acceptance rate is the metric your vendor can measure, which is why it is the metric you are offered. It is recorded at the moment of the keystroke — before the tests run, before review, before production — and nothing revises it when the code is reverted forty minutes later.

It is gamed in the direction that costs you most, and an agent does not need any intent to game it:

• Smaller suggestions accept better. Split one 40-line change into three 12-line ones and the numerator goes up while nothing else changes. An agent tuned for acceptance learns to propose the safe half of the work.
• Boilerplate accepts better than judgment. The suggestion that writes the getters is accepted; the one that questions the data model is not. A high acceptance rate is partly a measure of how boring the work was.
• Duplication accepts better than reuse. "Here is the validation logic, inline" needs no context to approve. "Import the shared validator and change its signature" needs three files of review. The metric prefers the first.
• A rejected suggestion is not a loss. Sometimes the useful output was the three seconds of thinking it prompted. Acceptance rate scores that as zero.

And the empirical demolition, again from METR: developers in the trial accepted fewer than 44% of AI generations — and were 19% slower overall. If acceptance rate were a productivity measure, that number would have told you something. It did not even get the sign right.

"Time saved", self-reported

This is the one that ends up in the board deck, and it is the one the evidence destroys most completely. Give it its own section.

The METR trial, precisely

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, by Joel Becker, Nate Rush, Beth Barnes and David Rein of Model Evaluation & Threat Research, arXiv:2507.09089, v2 revised 25 July 2025. Here is the whole design, because the design is the reason the result is worth anything:

The four numbers everyone should be able to recite:

Before starting, the developers forecast AI would cut completion time by 24%. After finishing, they estimated it had cut completion time by 20%. It had increased completion time by 19%. Economics experts (n = 34) had forecast 39% faster and machine-learning experts (n = 54) 38% faster.

The screen recordings show where the time went. With AI allowed, developers spent less of their time actively coding and less reading or searching for information, and more time prompting, waiting on generations (about 4% of time), reviewing and cleaning AI output (about 9% of time), and — the detail that will ring true to anyone who has watched their own session — more time idle.

What it does and does not generalise to

The paper is unusually disciplined about this: it contains an explicit table of claims it does not support. Reproduced faithfully, the first three are that the study does not show that AI systems fail to speed up most software developers, that it does not show anything about domains other than software, and that it does not show near-future AI will not speed up developers even in this exact setting. The authors also state plainly that their results are consistent with small greenfield projects or work in unfamiliar codebases seeing substantial speedup, and that two of the five factors they judge most likely to explain the slowdown — deep repository familiarity, and repository size and maturity — simply do not apply in many settings.

Concretely, then:

The tooling caveat is the one people reach for first, and it is legitimate — a year of harness improvements is a lot in this field, and it is the same reason a benchmark leaderboard tells you almost nothing about your repository. We took that apart separately in what agentic coding benchmarks actually measure. But notice the caveat only rescues the slowdown. It does nothing at all to the perception gap, which is a fact about human estimation under a plausible-looking assistant, and there is no version of better tooling that makes people better calibrated about it. If anything, a smoother agent should make the gap worse.

One thing we could not verify: the paper reports 95% confidence intervals using HC3 standard errors and notes that a hierarchical bootstrap yields somewhat wider ones, but the numeric interval around the headline 19% appears only inside Figure 1 and Figure 15, never in the body text. Reconstructing it from pixels would be exactly the sin this post is about. Treat 19% as a point estimate from 16 developers, which is precisely as much as it is.

DORA: what a survey of 5,000 people can and cannot tell you

The other primary source people cite is DORA's State of AI-assisted Software Development 2025, drawn from a global survey fielded 13 June to 21 July 2025 with responses from nearly 5,000 technology professionals plus more than 100 hours of qualitative data.

The adoption picture first, because it sets the baseline:

• 90% of respondents use AI as part of their work.
• More than 80% believe it has increased their productivity.
• 30% report little to no trust in the code AI generates.
• Median AI experience: 16 months. Median time interacting with AI on their most recent workday: two hours, which DORA notes is about a quarter of an eight-hour day.

Hold those first two next to METR. Over 80% of a 5,000-person sample believe AI increased their productivity; 16 developers who had just completed 246 randomly assigned tasks believed the same thing and were measurably wrong. The survey number is not evidence against the trial. It is the same instrument the trial showed to be unreliable, applied at scale.

Now the delivery findings, which are the interesting ones:

The 1.5% / 7.2% pair is DORA's 2024 estimate, restated verbatim in the 2025 report's chapter on AI's relationship to key outcomes. The 2025 report does not publish an equivalent percentage pair; it publishes standardised comparisons with 89% credible intervals, plotted, and every outcome in that chart sits between roughly 0 and 0.20 standard deviations. These are small associations. Anyone quoting DORA for a dramatic number is quoting the 2024 figures at you and probably not saying so.

DORA is also more careful than its readers. In a footnote to that chapter it explains that, unlike the previous year, it will speak "in terms of comparisons" rather than effects, quoting Regression and Other Stories to the effect that labelling such coefficients as effects is inappropriate without a great many assumptions, and that what is observed is an observational pattern. Everything above is self-reported, cross-sectional, and correlational. You cannot tell from it whether AI adoption destabilises delivery or whether teams with unstable delivery adopt AI more eagerly.

What survives all that hedging is the shape, and the shape has been stable across two reports: AI moves speed and stability in opposite directions unless something else in the system changes. DORA's own framing is that AI is an amplifier rather than a fix — it magnifies what a team already is. That is a claim about your delivery pipeline, not about your model choice, and it is the strongest argument in this whole post for measuring the stability side at all rather than only the speed side.

Four things you can measure in-house, and the trap in each

You do not have a research budget, you cannot randomise your production work for four months, and you should not try. Here is what is actually available, in rough order of signal per unit of effort.

1. Cycle time on comparable tasks

First commit to merge, on tasks of a similar kind. It is the closest in-house analogue of what METR measured and the only one of the four that responds inside a fortnight.

The trap is comparability, and it is severe. Nobody assigns the agent at random. You reach for it on the tasks you already believe it suits, which biases the comparison upward, and you avoid it on the gnarly ones, which biases it upward again. Then the tasks where you did use it and it went badly get quietly recoded in memory as "that was a hard one anyway". The fix is not more data. The fix is a coin flip, and it is the entire reason the protocol below asks for one.

Second trap: cycle time is a scalar and the thing it summarises is not. METR's recordings showed coding time falling and prompting, reviewing, waiting and idle time rising. A flat cycle time can conceal a large reallocation of how the hours were spent — which matters, because "less coding, more reviewing" is a different job with different fatigue and different failure modes.

2. Review burden

Reviewer minutes per merged change, and comment threads per change. If an agent is producing diffs that take longer to review than they took to write, the saving moved rather than existing.

The trap is the denominator. Measure per PR and an agent that splits work into more, smaller PRs looks like it lightened review while adding to it in aggregate. Measure per merged line and duplication flatters you. Reviewer minutes per task is the honest unit, and it is the one nobody instruments. A related trap in the other direction: review time can rise because the diff got bigger, not worse. Track diff size alongside, or you will read verbosity as risk.

3. Defect escape rate

The share of defects found after release rather than before it. This is the metric that answers the question everyone actually has, which is not "is it fast" but "is it fast in a way that comes back to us later".

The trap is the base rate. Escapes are rare, so the number is dominated by noise at team scale over any window shorter than a quarter. Worse, it is reflexively unstable: the moment people become suspicious of AI-written code, they review it harder, which raises the pre-production catch rate and lowers the escape rate for a reason that has nothing to do with the code being better. There is a second-order version of this trap in your test suite, where a generated test written against the implementation passes on a function whose boundary is wrong — we broke that one apart in letting an agent write your tests. A suite that agrees with the code is not a suite that caught anything.

4. Rework rate

Worth naming precisely, because DORA's definition is better than the folk one. DORA counts rework rate as the ratio of deployments that are unplanned but happen as a result of an incident in production, and pairs it with change fail rate as the two components of its software delivery instability factor. It is the cleanest available proxy for "we shipped something that came back".

The trap is deployment frequency. If you deploy twice a week, a rate computed over a fortnight has four data points and means nothing. And the rework you most want to see — the second and third attempt at the same change, before it ever merged — happens upstream of any deployment and never appears in this number at all. If you want that, count force-pushes and reopened PRs per task, and accept that it is a crude instrument.

A two-week protocol you can actually run

This will not produce a statistically significant answer and nothing that fits in two weeks will. METR needed 246 randomised tasks and still devotes an appendix to 21 things that might explain the result away. What this does produce is a calibration curve for your own team, which is the thing that is genuinely broken and the thing you can fix cheaply.

1. Define one eligible task type. "Bug fixes and small features in the main service, expected to take between one and six hours." Anything outside that is not in the study. Narrow beats representative here; you are trying to make two piles comparable, not to describe your whole workload.
2. Randomise per task, not per person. Coin flip at the moment the task is picked up — agent allowed or agent disallowed. Per-person assignment confounds the tool with the developer, and with two weeks of data you cannot afford a single confound. This is the step people skip, and skipping it makes everything downstream decorative.
3. Forecast before you flip. Write down two numbers: how long you think it will take with the agent, and how long without. Fifteen seconds per task. This is the whole point of the exercise. It is METR's key instrument, and it is what turns a fortnight of note-taking into a measurement of your judgment rather than of your tooling.
4. Log wall-clock, not calendar. Time actually spent on the task. A stopwatch, a timesheet, whatever you will actually use. Calendar time measures your meeting schedule.
5. Record spend per task. One line: tokens or dollars, from the session. See the next section for why this is the easiest honest number in the whole protocol.
6. At merge, record two more things: reviewer minutes, and whether the change needed a follow-up fix within 14 days. The second one will mostly be empty. Record it anyway; it becomes useful in the third month, not the first.
7. Read it as three questions, in this order. (a) How far apart are your forecasts and your actuals, and in which direction? (b) Did the two piles differ in median wall-clock — and if the difference is under about 20%, say "we could not tell" rather than picking a winner. (c) Did review minutes per task move, and did anything come back?

Resist two temptations. Do not compute a p-value on thirty tasks; you are not powered and the number will only be used to end an argument it cannot settle. And do not let the two piles drift apart — the moment somebody overrides a coin flip because "this one obviously needs the agent", write it down as an override and report the count, because that count is itself a finding.

If what you are choosing between is models rather than agent-versus-no-agent, the cheaper protocol is a fixed task set replayed from a clean checkout rather than a randomised live trial — we wrote that one up separately, and five tasks from your own git history is enough to kill a bad option.

The cost side, which is the one unambiguous number

Everything above is contested. Spend is not. Tokens are metered, priced and attributable, and it is the only figure in this discussion that does not depend on anyone's recollection — which is exactly why it is worth having, and slightly odd that so few teams attribute it to anything smaller than a developer-month.

The unit that matters is cost per accepted change, not cost per token and certainly not cost per seat. A model that is a fifth of the price but needs three attempts is not a fifth of the price, because each attempt resends the whole conversation. We worked the full arithmetic — subscription-versus-API break-even, where the tokens really go, what /usage does and does not report — in what an AI coding agent actually costs. For the purposes of this post you need one line of it: run /usage at the end of a session for a list-price figure, or export OpenTelemetry metrics if you want it continuously, and divide by tasks rather than by month.

Then use it to keep the rest of the discussion in proportion. The following is illustrative arithmetic on an assumed rate, not a measurement. Take a fully loaded developer day at $600 and a heavy agent user at, say, $13 of tokens a day. Tokens are about 2% of the day. A 19% swing in completion time — in either direction — is worth roughly $114 a day, close to nine times the token spend:

loaded developer day   $600
tokens per day          $13   ~2%
19% time swing         $114   ~9x tokens

Two conclusions, and they point opposite ways on purpose. First, optimising token spend while ignoring the time question is optimising the small number; if you have to choose what to instrument first, instrument time. Second, and less comfortable: because the time question is the expensive one and also the one you cannot resolve in a fortnight, "it is cheap, so who cares" is not an argument — the thing that would hurt you costs nine times what you are watching, and you are watching the cheap one because it is the one with a dashboard.

The short version

• Lines, acceptance rate and self-reported time saved are the three worst metrics available, and they are the three most likely to be handed to you. Each rewards an agent that produces more code than you needed.
• The METR trial found 19% slower against a 20% belief in faster, on 246 randomly assigned issues with 16 experienced developers in repositories they had worked in for five years. The slowdown may not read across to you. The 39-point perception gap does.
• DORA 2025 found higher throughput and higher instability together, across nearly 5,000 respondents — and disclaims causality in its own footnotes. The 1.5% / 7.2% pair people quote is the 2024 estimate.
• Measure cycle time on comparable tasks, reviewer minutes per task, defect escape rate and rework rate — and know which trap each one has before you present it.
• Randomise per task and forecast before you start. Without the coin flip you are measuring your task-selection instincts; without the forecast you learn nothing about the only failure mode the evidence has actually established.
• Track spend per task because it is the one number that cannot lie — then remember it is roughly a ninth of what a time regression would cost you.

Where Backgrind fits

Honestly: Backgrind will not tell you whether your agent is helping. It runs your own CLI, and that measurement is yours to run — this post is the protocol, not a product page.

There is one line in the METR data we do have a claim on, and it is a small one. When AI was allowed, the share of time developers spent idle went up. Some of that is thinking, and some of it is a person sitting in front of a terminal that is either generating or waiting for an answer to a permission prompt. Backgrind exists for the second kind: it floats the session over whatever you are actually doing and pings you when the agent needs a decision or is done, so blocked time stops being wall-clock you paid for. That is one term in one of the four metrics above, and we would rather claim it precisely than claim a productivity multiplier we have not measured. See what it does.

Frequently asked questions

Sources

All figures read from primary sources on 2026-08-13. Trial design, forecasts, 19% slowdown, sub-44% acceptance, activity labels, 47%-more-lines figure and the table of non-claims: Becker, Rush, Barnes and Rein, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, arXiv:2507.09089v2, 25 July 2025 (METR's summary). Adoption, trust, median usage time, the throughput/instability findings, the restated 2024 estimates, the five delivery metrics and the comparisons-not-effects footnote: DORA, State of AI-assisted Software Development 2025 (v.2025.2, survey fielded 13 June – 21 July 2025); publication list checked at dora.dev/research/publications. Copy/paste versus moved lines and the eight-fold duplicate-block increase: GitClear, AI Copilot Code Quality 2025 (v2025.2.5, 211M changed lines, Jan 2020 – Dec 2024).