← All posts

Guide

Letting an agent write your tests

Writing tests is the first serious job most people hand an agent. It feels like the safe one — tests do not ship to users, a bad test can be deleted, and the work is tedious enough that delegating it is pure upside. It is also the one place where a wrong answer is indistinguishable from a right one. Production code that is wrong eventually breaks something. A test that is wrong just sits there, green, being counted as evidence.

So we ran it. One small function, one agent, two different prompts, and then a deliberate campaign of breaking the implementation to see which tests noticed. The gap between the two prompts turned out to be the whole story.

What we ran

The subject was a seven-line function — apply a coupon to a cart subtotal, with an expiry check, a minimum-spend threshold, a percentage discount, and a cap:

export function applyCoupon(subtotalCents, coupon, now) {
  if (!coupon) return subtotalCents
  if (now > coupon.expiresAt) return subtotalCents
  if (subtotalCents <= coupon.minSpendCents) return subtotalCents
  const raw = Math.round((subtotalCents * coupon.percentOff) / 100)
  const discount = Math.min(raw, coupon.maxDiscountCents)
  return subtotalCents - discount
}

That implementation has one deliberate bug, and it is the kind that survives review. A separate SPEC.md says the coupon applies when the subtotal is at or above minSpendCents — spend fifty dollars, get the discount. The code says <=, so a cart of exactly fifty dollars is rejected. One character, at a boundary, in the branch nobody stares at.

Two headless runs of Claude Code 2.1.223 on Sonnet, each in its own empty directory, each told to use the built-in node:test runner so there was nothing to install:

• Run A — the normal prompt. The directory contained the implementation and nothing else. "Write unit tests for src/coupon.js… Aim for good coverage of the function. When you are done, run node --test and make sure everything passes." This is what almost everybody actually types.
• Run B — behaviour first. The directory contained SPEC.md and no implementation at all — not hidden, not discouraged, absent. "Write the tests from the spec alone… importing { applyCoupon } from '../src/coupon.js'… Cover every rule the spec states and every boundary it names explicitly." Then we copied the implementation in afterwards and ran the suite.

The test that agreed with the bug

Run A produced 17 tests. Run B produced 19. Against the buggy implementation, run A passed 17 of 17; run B passed 17 and failed 2. Here is run A's sixth test, name included, wrapped to fit:

test('returns subtotal unchanged when subtotal equals minSpendCents (boundary)', () => {
  const coupon = {
    percentOff: 10, minSpendCents: 5000, maxDiscountCents: 100000, expiresAt: 1000
  }
  assert.equal(applyCoupon(5000, coupon, 0), 5000)
})

The agent found the boundary. It labelled it a boundary. And then it asserted the wrong side of it, because the only source of truth in the room was the code. A few tests later it left this comment, which is the failure mode confessing in its own words:

// subtotal (0) is not > minSpendCents (0), so unchanged

That is not a test of behaviour. It is a paraphrase of line four. Nothing about it is careless — it is diligent, well-named, thorough work performed against the wrong oracle. Which is exactly why it survives a skim: a reviewer reading that test file sees the boundary case covered and moves on.

Run B's equivalent test asserted 5000 - 500 and went red on contact. That failure was the bug report. It is worth being precise about what happened, though: run B's other failure was its own fault. It built a coupon with a 200-cent cap, then computed its expected value as ten percent of the subtotal and forgot to apply the cap. One genuine find, one false alarm. Spec-first is not self-verifying — it is just pointed at the right target.

The failure modes, by name

The boundary case above is one of a small family. All of them produce a green suite; none of them produce evidence.

• The tautological assertion. The test recomputes the expected value using the same expression the implementation uses: assert.equal(applyCoupon(5001, c, 0), Math.round(5001 * 0.9)). That line is from run A, and it will pass for any rounding rule the implementation happens to use, because both sides move together. The tell is arithmetic in the expected value. A real expectation is a literal you were willing to write down.
• The mock of the thing under test. Ask for a unit test of a function with three dependencies and you often get all three mocked — and then, one refactor later, the function itself stubbed out somewhere in the setup. The suite now asserts that the mocks were called in the right order, which is a test of the test. The tell: a failing assertion whose message mentions a mock and never mentions a value your users would recognise.
• The snapshot of current output. An agent asked to "add tests" for a renderer or a serialiser will reach for toMatchSnapshot(), because it is the fastest way to produce something that passes. The first run writes the snapshot file from whatever the code does right now, bug included, and every future run defends it. Vitest and Jest both write a missing snapshot silently on a local run and only fail on a missing snapshot in CI — so the moment the bug gets frozen is precisely the moment nothing goes red.
• The unreachable assertion. A test whose assertion never executes cannot fail. We wrote three and ran them on Node v22.22.2: an assertion inside a callback that is never invoked, a loop body over an empty array, and an assertion inside a floating promise. All three reported ok. Node's runner did catch the third at file level, with an unusually honest message — "generated asynchronous activity after the test ended… would have caused the test to fail, but instead triggered an unhandledRejection event." Two of three passed in total silence. The countermeasures exist and cost one line: expect.assertions(2) or expect.hasAssertions() in Vitest and Jest.
• The test that pins the bug. The one we hit. Not a defect in the test, a defect in what it was pointed at.

Mutation testing: the check that actually works

Every failure mode above has the same shape — the suite is green and you cannot tell whether that means anything. Reading the tests does not settle it, because they read fine. Coverage does not settle it, for reasons we get to below. There is exactly one cheap procedure that does: break the implementation on purpose and see whether the suite goes red.

That is mutation testing. Change one operator, one constant, one guard clause — a "mutant" — and rerun the suite. If a test fails, the mutant is killed: some test genuinely depends on that behaviour. If the suite still passes, the mutant survived, and you have just learned that nothing in your suite checks the thing you broke. Your mutation score is the percentage killed.

We wrote nine mutants for the coupon function and ran both suites against all of them. First we fixed the implementation to match the spec, so both suites had a fair, green baseline. That fix immediately broke one test in each: run A's (boundary) test — the one that had pinned the bug — and run B's cap-arithmetic mistake. We quarantined those two and scored the rest.

Run A: 8 of 9. Run B: 9 of 9. Read the survivor carefully, because it is the point of the whole exercise. Even after we deleted the test that had asserted the bug, run A's suite still could not tell the difference between the corrected implementation and the broken one. Reintroduce the original defect today and that suite stays green. The bug was not merely missed; the suite is structurally blind to it, because it was written by looking at it.

Eight of nine is not a bad score in absolute terms — the two suites are near-identical on seven behaviours, and both agents did competent work. The difference is entirely concentrated in the one place where the code and the intent disagreed, which is the only place a test was ever going to earn its keep.

Doing it without a tool

You do not need to install anything to get most of this. Open the file the agent just tested, break one thing, run the suite, put it back. Five or six mutations on the branch you actually care about takes ten minutes and answers the only question that matters. The mutations worth trying first, in order of how often they survive: comparison operators at a boundary, a deleted guard clause, a swapped constant, a flipped sign, and a removed Math.min/Math.max clamp.

When you want it automated and repeatable, the tooling is mature and boring in the good way: StrykerJS (@stryker-mutator/core 9.6.1) for JavaScript and TypeScript, with runner plugins for Vitest, Jest, Mocha, Jasmine, Karma, Cucumber and Tap; PIT 1.25.9 for the JVM; mutmut 3.7.0 for Python. Two Stryker settings matter more than the rest on a real repo: mutate, the glob that decides which files get mutated at all, and incremental (off by default), which caches results between runs so you are only paying for what changed. Stryker's default score thresholds are 80 for high, 60 for low, and no break value — meaning out of the box it reports a score and fails nothing. Pointing it at one directory after an agent writes tests there is a better use of it than a repo-wide gate.

Coverage rewards exactly the wrong thing

Here is the number that should end the argument. Run A's suite, on the implementation it was written against, reported:

# file            | line % | branch % | funcs % | uncovered lines
#  coupon.js      | 100.00 |   100.00 |  100.00 |

One hundred percent line coverage, one hundred percent branch coverage, one hundred percent function coverage — on a function whose min-spend comparison was wrong, from a suite that asserted the wrong behaviour at that exact boundary. Every line ran. Nothing was verified.

This is not a quirk of our example; it is what the metric measures. Coverage asks "did this line execute during the test run?" It has no opinion on whether anything was asserted afterwards, or whether what was asserted was right. Which means coverage is trivially satisfiable by the exact behaviour you least want from an agent: call every branch, assert whatever came back. Ask for "good coverage" — as we literally did in run A — and you are asking for lines to be executed. The agent complied perfectly.

Mutation score is harder to game because it is defined by consequences rather than by execution. A test that mirrors the implementation cannot kill a mutant of that implementation, no matter how many lines it touches. That property is the whole reason to prefer it. Use coverage the way it works — as a floor, to find files nobody tested at all — and use mutation score where it matters, on the handful of modules where being wrong is expensive.

How to prompt for tests worth having

None of this argues for writing tests by hand. It argues for changing what you put in front of the agent, which costs a few sentences.

• Give it the behaviour, not the code. The whole difference between our two runs was which artefact was in the room. You rarely have a SPEC.md lying around, but you almost always have something: the issue text, the PR description, the docstring, the acceptance criteria, the paragraph you would say out loud to a colleague. Paste that and say "test these rules." If you can arrange for the implementation to be genuinely out of reach — a separate directory, a read-restricted subagent given the spec and the import path — do it, because "don't look at the implementation" is an instruction and an absent file is a fact.
• For a bug fix, demand the failing test first. "Write a test that reproduces this bug. Run it. Show me it failing. Do not touch the implementation yet." A test that has been observed failing for the right reason has proved it can turn red — you have done one mutation, on the mutant that actually occurred in production. Then fix, and watch it go green. Skipping the red step is how a suite fills up with tests that were never capable of failing.
• Ask for the literal, not the formula. Ban arithmetic in expected values: no Math.round(x * 0.9), no re-deriving the answer. "Every expected value must be a literal you can justify from the spec." This one line kills the tautological-assertion mode outright.
• For legacy code with no spec, name what you are doing. Characterisation tests are the honest version of "write tests from the implementation" — you are deliberately recording current behaviour so a refactor cannot change it silently, and you are not claiming it is correct. Ask for exactly that, and ask for the file to say so at the top. The failure is not writing them; the failure is writing them and then filing them under "verified."
• Make the check part of the task. "After the tests pass, break the implementation in three places one at a time, confirm the suite fails each time, and restore it. Report anything that stayed green." Agents are good at this — it is mechanical, verifiable work — and it turns the review question from "do these tests look right?" into "here is the list of things they provably catch."

Two habits that make the rest work

First, the second-reviewer logic applies here too: an agent that wrote the implementation is the worst candidate to write its tests, for the same reason it is the worst candidate to review them — the argument is in AI code review in 2026. A different session, at minimum, and ideally one that never saw the code. Second, a mutation loop means running the test command dozens of times, and approving each run by hand ruins it. Scope a rule so the test command runs unattended in that repo while everything else still asks — the mechanics are in pre-approving what your agent is allowed to do.

What this experiment does not prove

One function, one model, one afternoon. Nine hand-written mutants on seven lines of code is a microscope, not a survey — and we chose the bug, which means we chose a case where spec-first wins. A different function with a different defect could easily produce a different gap. Neither run was incompetent; both suites were well organised, well named, and would pass code review.

What it does establish is the mechanism, and the mechanism is not luck. An agent given only the implementation has no way to distinguish a bug from a decision, because the two are the same bytes. That is not a model capability problem and no amount of scaling fixes it. Anything you would not have known without the spec is exactly what the tests will not check.

Where Backgrind fits

Backgrind does not write your tests — your own agent does that. Backgrind is the always-on-top window it runs in, with a tab per session and a background daemon that keeps the work alive when the window is closed. It matters for this particular job because a mutation loop is slow and dull in exactly the way you should not sit and watch: nine breaks, nine test runs, one interruption somewhere in the middle when the agent wants permission for something it has not been pre-approved for. That tab pings, you answer from the toast, and the rest keeps grinding while you are looking at something else. It is a GUI for your agents, not another agent. See the loop in the live demo.

Frequently asked questions

Sources

The experiment was run on 7 August 2026 on macOS, Node v22.22.2, using the built-in node --test runner and node --test --experimental-test-coverage. Both test suites were generated by Claude Code 2.1.223 in headless mode (claude -p) on Sonnet, in separate empty directories; the prompts are quoted above in full apart from the boilerplate about the runner. The nine mutants were applied by a short script that rewrites one substring in the source, reruns the suite, and restores it; a mutant counts as killed when the suite exits non-zero. Baselines were confirmed green before scoring, and the two quarantined tests are named in the text. This is a single seven-line function — treat the 8/9 and 9/9 as an illustration of a mechanism, not a benchmark. Versions read on 7 August 2026: @stryker-mutator/core 9.6.1 and vitest 4.1.10 from the npm registry; mutmut 3.7.0 from PyPI (uploaded 31 July 2026); PIT 1.25.9 from the pitest releases (published 4 August 2026). Stryker's runner plugins, the mutate and incremental options and the default thresholds of high 80 / low 60 / break null: StrykerJS configuration. expect.assertions(), expect.hasAssertions() and snapshot behaviour: Vitest expect API and Vitest snapshot guide, which documents that snapshots are not written when process.env.CI is truthy and that missing snapshots fail the run there.