Explainer

Agentic coding benchmarks in 2026, decoded: what each one actually measures

The short version: in 2026 the four benchmarks people quote measure four different things, and only two of them still discriminate between frontier models. SWE-bench Verified is effectively retired — its own biggest publisher stopped reporting it in February 2026 over broken tests and contamination. Terminal-Bench measures the harness at least as much as the model. Frontier-Bench v0.1 and ARC-AGI-3 are the two that still have headroom, which is exactly why Anthropic's Claude Opus 5 announcement led with Frontier-Bench and published no SWE-bench number at all. And none of the four tells you how a model will behave in your repo, because none of them contain your repo.

The four benchmarks, side by side

SWE-bench: the benchmark that outlived its own signal

SWE-bench (ICLR 2024) was a genuinely good idea: 2,294 task instances built from real GitHub issues and their merged pull requests across 12 popular Python repositories, graded by the project's own tests. At publication, Claude 2 with a BM25 retriever resolved 1.96% of them — the anchor every later "jump" should be read against. Verified, released August 2024, is the 500-instance subset OpenAI built with the SWE-bench authors after 93 experienced Python developers annotated 1,699 sampled instances to screen out underspecified issues and unfair tests. About 22% of the original, and the number that ended up on every model card for two years.

Then it broke. In February 2026 OpenAI published its reasoning for dropping the benchmark: an audit of 138 problems its o3 model consistently failed found 59.4% contained flawed test cases — either too narrow (enforcing implementation details the issue never described) or too wide (testing behavior nobody asked for). Contamination analysis showed every frontier model tested could reproduce exact gold patches and verbatim problem details. The primary post blocks automated fetching, so treat those precise figures as consistently-reported secondary rather than something we read off the source.

The independent memorization evidence is cleaner. A 2026 workshop paper asked a blunt question: given the issue text alone, with no file structure, can a model name the files to edit? On SWE-bench Verified, models get 63–65% right. On BeetleBox, a held-out equivalent, 12%. Hand them the file structure and it becomes 73–76% versus 18–21%. A five-fold gap is not reasoning transferring; it is recall.

The successor did not save it. SWE-bench Pro (Scale AI, Sept 2025) deliberately sourced its 731 public tasks from strong-copyleft repos as a legal deterrent against training-set inclusion, and top models scored ~23% at launch versus 70%+ on Verified. But on 8 July 2026 OpenAI retracted its own February recommendation to switch: roughly 30% of the public split is broken, with 200 tasks flagged by an automated pipeline and 249 by five human annotators — figures we have via secondary reporting, not off the OpenAI post itself, which blocks fetching. Dominant failure mode, again: hidden tests stricter than the task text. Any current SWE-bench Pro leaderboard you see is scored on that split.

Terminal-Bench: you are benchmarking the harness

Terminal-Bench moved the goalposts usefully — instead of a patch that makes pytest pass, it asks an agent to do end-to-end work in a real terminal. v1.0 shipped 19 May 2025 with 80 tasks; v2.0 has 89. The interesting release is v2.1 (6 May 2026), which patched 28 of those 89 tasks: 9 for external dependencies, 8 for resource-budget mismatches, and 11 for outright misspecification, including a query-optimize task whose grader expected Spark SQL while the instructions asked for PostgreSQL.

Fixing the benchmark alone moved Claude Code with Opus 4.6 up 12.1 points. Per-task swings ran from +84.3% to −18.6%. If you saw a model "improve" across that window, some of the improvement was the ruler.

The harness effect is the bigger trap. On the official 2.1 leaderboard — whose entries are submitted by the labs themselves, not independently re-run — the same model scores differently depending on the scaffold driving it:

A 5.1-point scaffold spread is larger than the gap between the #1 and #3 models. A "Terminal-Bench score" with no version and no harness attached is therefore not a fact. This is why independent re-measurement matters: Artificial Analysis runs v2.1 itself on a uniform Terminus 2 harness in an e2b sandbox, pass@1 averaged over three repeats, and gets GPT-5.6 Sol (xhigh) 89.5%, Claude Opus 5 89.1%, GPT-5.6 Sol (max) 88.0%. Same ruler for everyone beats a higher number.

Frontier-Bench: built because the last one saturated

Frontier-Bench v0.1 launched around 23 July 2026 from the Terminal-Bench and Harbor team (Stanford × Laude Institute, with Snorkel AI as task author and data partner): 74 tasks across seven domains — software, ML, science, operations, security, hardware, media — over artifacts like databases, RTL designs, formal proofs, VM images, binaries and CAD files. Launch scores: GPT-5.6 Sol 34.4%, Fable 5 33.8%, Opus 4.8 21.1%.

The stated reason for existing is discrimination. On Terminal-Bench 2.1 the top models cluster in a compressed 74–84% band; Frontier-Bench separates Fable 5 from Opus 4.8 by 12.7 points against the 4.9 the announcement reports for the same pair there — a 2.6x wider gap on the same two models, on the benchmark authors' own numbers. The methodology is also the most defensive yet: tasks are validated with both oracle-agent runs and deliberate "cheating agent" runs, they carry a canary GUID marking them train-excluded, and they are semantically versioned so old trials can be re-graded when verifiers improve.

Good design — and still self-reported when a vendor runs it. Anthropic's Opus 5 announcement (24 July 2026, $5/$25 per MTok, unchanged from Opus 4.8) leads with Frontier-Bench and carries one methodology footnote: an internal run of v0.1 on the mini-SWE-agent harness and a GKE backend, mean reward over 5 attempts, with Opus 4.8 serving as fallback on safety-classifier refusals. Its 43.5% is tracked as zero verified, one self-reported — different harness, backend and attempt budget than the launch table, so not directly comparable to the 34.4% and 33.8% above. Note also what is absent: no SWE-bench and no Terminal-Bench number anywhere in the announcement or the docs comparison table. Aggregators quoting "96% SWE-bench Verified" for Opus 5 are quoting each other, not Anthropic. See what actually changed in Opus 5.

ARC-AGI: the one still unsolved

ARC-AGI-2 launched 24 March 2025 with 120 semi-private eval tasks. At launch pure LLMs scored 0%, o1-pro 1%, o3-preview-low 4%, against a ~60% average across 400+ human testers. By July 2026 the scores ARC Prize published were GPT-5.6 Sol (max) 92.5% and Claude Opus 5 (max) 90.4%. Zero to 92.5% in sixteen months — the cleanest illustration in the field that a static benchmark's useful lifespan is now about a year.

ARC-AGI-3, the interactive agentic version, has not budged the same way. As of 24 July 2026, Claude Opus 5 at high effort set a new high of 30.16%, clearing five public demo environments no model had beaten before; GPT-5.6 Sol at max effort managed 7.78%. Whatever "agency over many steps in an unfamiliar environment" is, it is not solved. Read it with ARC Prize's own disclosure attached: they state plainly that they do not verify submissions by default, cap eval runs at $10,000, and reimburse only up to $2,500 for reproductions.

Why none of this predicts your repo

Every benchmark above evaluates a model on code it did not have to live with. Your repo has conventions no issue text explains, a test suite with its own pathologies, and a definition of "correct" that includes not waking the on-call engineer. The memorization data makes the point sharply: on a benchmark the model has probably seen, it identifies the right files 63–65% of the time from the issue text alone; on one it has not, 12%. Your codebase is in the 12% category by construction.

Cost is the other axis benchmarks ignore entirely. A model that scores two points higher and burns three times the tokens on a 40-turn agent loop is a worse choice for most work — run the numbers in the agent cost calculator and check current rates in the model pricing tracker before you switch anything.

The protocol that actually works

It takes an afternoon and it beats every leaderboard for your purposes.

1. Pick five tasks from your own git history where you already know the good answer: a bug fix, a refactor, a dependency migration, a test-coverage gap, and one genuinely hard design call. Real merged commits, so you have ground truth.
2. Freeze the conditions. Same clean checkout, same prompt, same tool permissions, same CLAUDE.md for every candidate — changing the scaffold between runs reproduces the exact harness confound that makes leaderboards unreadable.
3. Log cost and wall-clock time per task, not just pass/fail. That is where the frontier tax shows up.
4. Score blind, a day later. Strip model names from the diffs, then rank against your merged commit. The day's gap kills the recency bias that makes whichever model you ran last look sharper.
5. Re-run when the model changes. Point releases move real behavior, and five tasks are cheap to replay.

Five tasks will not give you a statistically defensible number. It will reliably kill a bad option, which is the decision you were actually trying to make. The longer version is in how to compare AI models on your own codebase; if you want a second opinion on one hard call rather than a ranking, asking several models the same question and reading the spread is the cheaper instrument.

Frequently asked questions

Where Backgrind fits

Backgrind is not a model and has no benchmark score. It is a desktop overlay for macOS and Windows that PTY-wraps the CLI you already pay for — Claude Code, Cursor, Codex — or a hosted model (Grindy) if you would rather not manage keys. It is an always-on-top window that stays visible over other apps, including borderless-fullscreen games, and it pings you when an agent needs a decision or finishes. In BYO-CLI mode your agent's content never touches our servers.

For the protocol above, the useful part is running the five tasks in parallel tabs rather than serially and leaving them running while you do something else. For the single hard design call, the Fusion council seats your own CLIs on one question, read-only in the repo, and a chairman judges their answers into one verdict — pinging you when they split, which is the case worth your attention.

Sources

Frontier-Bench v0.1 design and launch scores: frontierbench.ai — announcement. Terminal-Bench task repairs and score movement: tbench.ai — Terminal-Bench 2.1; harness spreads from the official 2.1 leaderboard. Uniform-harness re-measurement: Artificial Analysis — Terminal-Bench v2.1. ARC-AGI-3 results and verification policy: ARC Prize — Claude Opus 5 and ARC Prize — GPT-5.6. Memorization gap: "Does SWE-Bench-Verified Test Agent Ability or Model Memory?". SWE-bench Pro construction: arXiv 2509.16941 (Scale AI). Original SWE-bench: ICLR 2024 paper; Verified subset methodology. Opus 5 benchmark claims (self-reported, internal harness): Anthropic — Claude Opus 5; on eval methodology generally, Anthropic — demystifying evals for AI agents. OpenAI's retraction of SWE-bench Pro as a recommended replacement is reported secondary via Techmeme, 8 July 2026; openai.com blocks direct fetching, so its February and July posts are cited as reported rather than read.