← All posts

Guide

Running a coding agent in CI: the walls, in the order you hit them

Everything you know about driving a coding agent assumes you are there. You watch the diff go by, you answer the permission prompt, you say "no, not that file" when it heads somewhere wrong. CI removes all three at once. The agent gets one shot, on a machine with no browser, no keychain and nobody to ask.

That changes which failures you get. On a laptop, a badly configured agent stops and waits for you. In CI it does not stop — it finishes, reports success, and hands you a green check over an empty diff. We measured exactly that, and it cost thirty-eight cents.

Headless is a different tool wearing the same name

The interactive claude you use every day and the one that belongs in a pipeline share a binary and very little else. Add -p (or --print) and the agent stops being a REPL: it takes one prompt, runs to completion, writes a result to stdout and exits.

claude -p "Fix the failing test" --allowedTools "Read,Edit,Bash(npm test)"

Three properties matter once there is no human attached. It exits 0 on success and non-zero when the run fails, so a shell can branch on it. With --output-format json the response is a single object with the text in result and the metadata your workflow needs to make decisions — total_cost_usd, num_turns, duration_ms, session_id, is_error, and a permission_denials array that turns out to be the most important field in the whole payload. And --max-turns caps how many times it can loop before giving up.

There is also --bare, which the docs describe as "the recommended mode for scripted and SDK calls" and which will become the default for -p in a future release. It skips auto-discovery of hooks, skills, plugins, MCP servers, auto memory and CLAUDE.md. The argument for it in CI is reproducibility: "a hook in a teammate's ~/.claude or an MCP server in the project's .mcp.json won't run, because bare mode never reads them." Hold onto that flag. It is about to cause the first problem.

If what you actually want is a long-running agent you can still reach, that is a different problem with a different answer — see running Claude Code in the background. CI is the case where you genuinely cannot be reached, and everything below follows from that.

Wall one: authentication on a machine with no browser

This is where everyone stops first, and the reason is mundane. Your laptop is authenticated because at some point a browser opened and you clicked a button. A runner is a fresh VM that will be destroyed in fifteen minutes. There is no browser, no keychain, no /login.

Here is what that looks like, run in a clean container with no credentials at all:

$ claude -p "say hi" --output-format json ; echo "exit=$?"
{"is_error":true,"terminal_reason":"api_error","subtype":"success",
 "total_cost_usd":0,"result":"Not logged in · Please run /login", ...}
exit=1

Note the shape of that object, because it recurs: is_error is true while subtype is "success". If you gate your workflow on .subtype you will pass a run that never contacted the API. Gate on .is_error.

You have three real options for getting a credential onto the runner.

• ANTHROPIC_API_KEY — a key from the Claude Console, billed as API usage. This is the one that works across an organization, because it is not tied to a person. Store it as a repository or organization Actions secret.
• CLAUDE_CODE_OAUTH_TOKEN — a long-lived token from claude setup-token, available on Pro, Max, Team and Enterprise plans, which bills against that person's subscription rather than API credit. Cheap, and tempting for exactly that reason. The docs are explicit that it is "tied to the subscription of the person who ran claude setup-token", so it is a poor fit for anything shared.
• OIDC workload identity federation — no long-lived secret in the repository at all. The workflow's GitHub OpenID Connect token is exchanged for API access through a Console service account, configured with anthropic_federation_rule_id and anthropic_organization_id. It needs id-token: write on the job. This is the right answer for an organization that does not want a static key sitting in Actions secrets, and it is also how the Bedrock, Vertex and Foundry paths authenticate.

The combination that quietly does not work

Two pieces of official advice are each correct and do not compose. The headless docs recommend --bare for CI. The GitHub Actions docs offer CLAUDE_CODE_OAUTH_TOKEN as an authentication option. But bare mode, in the docs' own words, "never reads OAuth credentials or the system keychain."

We tested whether that includes the environment variable, using a deliberately invalid token — a 401 would prove the token was read and sent, while a login error proves it was ignored:

The first row never reached the API. So if you follow the recommendation to use --bare and you authenticate with a subscription token, your pipeline fails with an instruction to run /login — on a machine where that is impossible, with a valid token sitting right there in the environment. With --bare, use an API key (or an apiKeyHelper in --settings, or a cloud provider's own credentials). If you need the subscription token, drop --bare and accept that the run will read whatever hooks and MCP config the checkout contains — which, on second thought, is its own reason to prefer the API key.

Wall two: nobody is there to approve anything

On a laptop, permissions are a conversation. The agent proposes, you approve, and an ambiguous case just waits. Take the human away and "waits" becomes "hangs until the six-hour job timeout." So the question is not whether to prompt. It is what happens to the calls that would have prompted.

--permission-mode dontAsk is the answer built for this: every tool call that would otherwise prompt is auto-denied. Claude runs only what matches your allow rules, the read-only Bash command set, and anything a PreToolUse hook approves. The session never waits for input. That is the correct default for a pipeline — and it fails in a way that is much easier to miss than a hang.

We gave a small repository one genuinely failing test — a slugify() that lowercases and dashes correctly but never implements the 40-character word-boundary truncation the fourth test asserts — and ran four arms of the same prompt: "The test suite has one failing test. Run it, find the cause, and fix the source so all four tests pass. Do not change the test file."

Read the first row slowly. Exit code 0. is_error: false. subtype: "success". Eight turns, fifty-four seconds, thirty-eight cents — and the working tree was byte-identical to where it started. The agent had been denied npm test, denied node --test, and denied the Edit that contained the correct fix. It diagnosed the bug perfectly from reading the file and wrote this into its final message:

I'm blocked from completing this: both `Bash` and `Edit` are denied in this
session's permission mode, so I can neither run `npm test` nor modify
`slugify.js`. I stopped rather than trying to route around the denial.

That is exemplary behaviour by the agent and a catastrophe for your pipeline, because -p reports the run as a success. It completed. It just was not permitted to do anything. In a real workflow the next step finds no changes, skips the commit, and the job goes green. Nobody looks at a green job.

The fix is one line, and it is the reason permission_denials is the field that matters:

denied=$(jq -r '.permission_denials | length' agent.json)
if [ "$denied" -ne 0 ]; then
  echo "::error::agent hit $denied permission denials — the allowlist is wrong"
  jq -r '.permission_denials[].tool_name' agent.json
  exit 1
fi

What to pre-allow, and the trap in the second row

The second arm got the allowlist "Read,Edit,Bash(npm test)" and fixed the bug in nine turns. It also recorded one denial, and the denial is worth the whole section: the agent's first attempt was a Write call, not an Edit call. Edit and Write are separate tools, and allowing one does not allow the other. Here it recovered by falling back to Edit. A less lucky task — creating a new file, say — would have died there, exited 0, and told you nothing unless you were reading permission_denials.

A workable starting allowlist for a "fix the thing and open a PR" job:

• Allow Read, Edit, Write, Glob and Grep. The agent has to be able to see the repository and change it. Read-only Bash commands are already permitted by dontAsk's built-in set.
• Allow exactly the build and test commands, by prefix. Bash(npm test), Bash(npm run lint). The rule syntax uses a trailing space-star for prefix matching, and the space is load-bearing: Bash(git diff *) matches any git diff, while Bash(git diff*) also matches git diff-index.
• Do not allow git at all. Let the workflow commit and push, not the agent. The agent's job is to change files; git is the workflow's job, and keeping that boundary is what makes the diff reviewable.
• Do not allow WebFetch or WebSearch. A CI job with an egress path and an agent reading arbitrary pages is a much larger surface than the task needs.
• Deny beats allow, everywhere. If a settings file gives you deny rules, they still apply — deny rules and explicit ask rules are enforced in every mode, including the bypass one.

The general shape of this — decide up front, so nothing has to be decided mid-run — is the same argument as pre-approval policies on a laptop. CI just removes the escape hatch: there, an unmatched call escalates to you; here it is simply denied.

Why --dangerously-skip-permissions is a different calculation here

The honest version: the risk genuinely is lower in CI than on your laptop. A runner is ephemeral and rebuilt for every job. It has no home directory worth losing, no SSH keys, no ~/.aws, no eight years of side projects. The worst local damage is undone by the VM being destroyed nineteen minutes later. That is a real difference and it is why so many CI examples reach for the flag.

It is not zero, though, and the reasons are specific to CI rather than general nerves:

• The runner holds a checkout with push rights. The token is in the environment, and actions/checkout persists credentials by default so git push just works. "Ephemeral" describes the filesystem, not the repository.
• The prompt may contain someone else's text. The moment your trigger is an issue body or a PR comment, an untrusted string is being handed to an agent that can run shell commands. bypassPermissions "offers no protection against prompt injection or unintended actions" — that is the docs' own sentence, and CI is where it stops being theoretical. Our security review questions post covers the rest of that surface.
• You lose the audit trail. dontAsk gives you a machine-readable list of everything the agent tried and was refused. Bypass mode gives you nothing to gate on, because there is nothing to refuse.

There is also a practical reason it may not start at all. The flag is blocked under root, and a container: job runs as root by default — the same refusal, for the same reason, that bites when you run a coding agent inside a container. On a plain runs-on: ubuntu-latest the job runs as the non-root runner user, so it starts; in a container job it does not:

# in a container job (root)
$ claude --dangerously-skip-permissions -p "say hi"
--dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons
exit=1

Use dontAsk with an explicit allowlist. You get the same never-blocks property, plus a list of what it refused, plus a run that starts wherever you put it.

The workflow shape, and the token permissions it needs

The job is four moves: check out, run the agent, verify independently, open a PR. The third one is not optional. The agent telling you the tests pass and the tests passing are different claims, and only one of them is checkable.

Two things about the token before the YAML, because they are where this breaks. First, GITHUB_TOKEN's default is restrictive: on a new personal repository it "only has read access for the contents and packages scopes." Second, and less obvious — "if you specify the access for any of these permissions, all of those that are not specified are set to none." The permissions: block is a complete declaration, not a set of additions.

name: Agent fix

on:
  workflow_dispatch:
    inputs:
      task:
        description: One bounded, well-specified job
        required: true

permissions:
  contents: write
  pull-requests: write

concurrency:
  group: agent-fix-${{ github.ref }}
  cancel-in-progress: true

jobs:
  fix:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
      - run: npm ci
      - run: npm install -g @anthropic-ai/claude-code

      - name: Run the agent
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          TASK: ${{ inputs.task }}
        run: |
          claude --bare -p "$TASK" \
            --permission-mode dontAsk \
            --allowedTools "Read,Edit,Write,Bash(npm test),Bash(npm run lint)" \
            --max-turns 15 \
            --output-format json > "$RUNNER_TEMP/agent.json"

      - name: Gate on the run, not on the exit code
        run: |
          jq -r '.result' "$RUNNER_TEMP/agent.json" >> "$GITHUB_STEP_SUMMARY"
          echo "cost: \$$(jq -r '.total_cost_usd' "$RUNNER_TEMP/agent.json")" >> "$GITHUB_STEP_SUMMARY"
          test "$(jq -r '.is_error' "$RUNNER_TEMP/agent.json")" = false
          denied=$(jq -r '.permission_denials | length' "$RUNNER_TEMP/agent.json")
          if [ "$denied" -ne 0 ]; then
            echo "::error::agent hit $denied permission denials — the allowlist is wrong"
            jq -r '.permission_denials[].tool_name' "$RUNNER_TEMP/agent.json"
            exit 1
          fi

      - name: Refuse changes to CI config
        run: |
          if ! git diff --quiet -- .github/; then
            echo "::error::agent modified .github/ — refusing to open a PR"
            exit 1
          fi

      - name: Verify independently
        run: npm test

      - name: Open a pull request
        env:
          GH_TOKEN: ${{ github.token }}
          TASK: ${{ inputs.task }}
        run: |
          git diff --quiet && echo "no changes" && exit 0
          branch="agent/${{ github.run_id }}"
          git config user.name "agent[bot]"
          git config user.email "agent[bot]@users.noreply.github.com"
          git checkout -b "$branch"
          git commit -am "Agent: $TASK"
          git push origin "$branch"
          gh pr create --base "${{ github.ref_name }}" --head "$branch" \
            --title "Agent: $TASK" --body "Opened by run ${{ github.run_id }}. Review before merging."

Four things in there that are not decoration

• The prompt goes through env:, never inline. Writing claude -p "${{ inputs.task }}" directly inside a run: block splices attacker-controlled text into a shell command before bash ever sees it. Bind it to TASK and reference "$TASK". This matters far more once the trigger is an issue comment rather than a manual dispatch.
• The .github/ guard. An agent asked to "fix CI" will edit CI. GITHUB_TOKEN has no workflows permission, so a push touching .github/workflows/ is rejected outright — you get a failure at git push after the model has already been paid. Catching it before the push turns a confusing error into a clear one. (This rejection is widely reported but is not in GitHub's own docs, so treat the exact message as community-attested rather than specified.)
• timeout-minutes: 15. The default is 360, which is also the hard ceiling for a GitHub-hosted job. Six hours of a stuck agent is not a scenario you want to discover from a bill.
• concurrency with cancel-in-progress. Two agents on the same branch produce two PRs that conflict with each other. One at a time.

The pull request itself has two more gotchas

"Allow GitHub Actions to create and approve pull requests" is off by default on new personal repositories, under Settings → Actions → General → Workflow permissions. Without it, the create call returns 403 no matter what your permissions: block says — both halves are required. Organizations and enterprises can disable it above you, in which case the repository checkbox is simply greyed out.

And CI on the agent's PR may not run by itself. The long-standing rule is that "events triggered by the GITHUB_TOKEN will not create a new workflow run" — which exists to stop recursion. That rule has recently been softened on GitHub.com specifically for pull requests: opened, synchronize and reopened now do create runs, but in an approval-required state, with a banner in the merge box until someone with write access clicks Approve workflows to run. Other activity types still create nothing, push events from the token still do nothing, and GitHub Enterprise Server has not received the change. If you need the PR's CI to start unattended, use a GitHub App token via actions/create-github-app-token instead of GITHUB_TOKEN.

If you would rather not own any of this, anthropics/claude-code-action@v1 wraps the same CLI and is the documented path for @claude mentions and issue-to-PR flows. Its published example asks for contents: write, pull-requests: write, issues: write, id-token: write and actions: read — the id-token entry is required for the action's default GitHub App authentication, and actions: read lets Claude read CI results. Everything above about allowlists and turn caps still applies; you pass them through claude_args.

Cost and time, because a runaway agent bills silently

A stalled agent on your laptop is free and annoying. A stalled agent in CI is neither. Nobody is watching the tab, so the only things standing between you and a surprise are the caps you set before the run.

Two meters run at once, and they are wildly different sizes.

• Runner minutes. A standard ubuntu-latest (2-core Linux) minute is $0.006 on a private repository, rounded up to the whole minute per job. Public repositories on standard runners are free. Free and Team plans include 2,000 and 3,000 private minutes a month respectively. Our 15-minute cap is worth about nine cents.
• Model tokens. The successful arm above — one bug, one file, forty-one seconds — cost $0.376. That is roughly four times the entire runner budget for the job it ran in.

So the runner is the cheap part, and every cost control that matters is a control on the agent. Which brings up the least intuitive number in the table: the arm that changed nothing cost slightly more than the arm that fixed the bug — $0.383 against $0.376. Being blocked is not cheaper than working. The denied agent spent its turns re-reading the file and reasoning about why it could not proceed, which bills exactly like progress.

The caps worth setting, in order of how much they save you:

• --max-turns. The hard stop on the agent loop. Set it, but set it honestly: our --max-turns 2 arm exited 1 with subtype: "error_max_turns" and errors: ["Reached maximum number of turns (2)"] — and still billed $0.191 for the work it did before the cap. A cap that is too low buys you a failed job at half price, not a free one. Fifteen is a reasonable starting point for a single-file fix.
• timeout-minutes. Independent of turns, because a single turn can hang. This is the one that catches the failure mode --max-turns cannot.
• Read total_cost_usd and put it in the job summary. It is a client-side estimate rather than your bill, but it is the number that shows you a task drifting from twenty cents to two dollars over a month of runs. Writing it to $GITHUB_STEP_SUMMARY costs one line and makes the trend visible without opening a dashboard.
• Pin the model. --model in claude_args or on the CLI. Letting a scheduled job inherit whatever the default becomes is how a cost profile changes without a commit.

What is actually worth doing in CI

The scope question is the one most posts skip, and it has a clean answer: CI suits work where the agent needs no judgment and the definition of done is checkable by a machine. That is a narrower set than it sounds, and the tasks inside it are genuinely valuable.

Good fits.

• A bounded fix with a failing test attached. The test is both the specification and the acceptance check. This is the shape our experiment used, and it is the shape that works — but see debugging with an agent for why a test asserting the symptom rather than the mechanism is satisfiable by suppressing it.
• The mechanical tail of a migration. Once a codemod has done the deterministic 80%, the remaining files that need small judgment calls are a good batch job — one PR per batch, each independently reviewable.
• Regeneration and hygiene. Types from a schema, a changelog from merged PRs, docs from source. Repeatable, verifiable, boring — the exact profile you want unattended.
• Advisory review. Piping git diff into a headless run and posting findings costs one API call and blocks nothing, because the output is a comment rather than a commit.

Bad fits, and the reason is always the same.

• Anything whose first honest step is "it depends." An agent that could ask a clarifying question will ask one. An agent in CI cannot, so it picks an interpretation and commits to it with total confidence. You do not get a question; you get a plausible wrong PR.
• Design and architecture. "Refactor the auth module" has no machine-checkable definition of done. The tests passing afterwards tells you almost nothing about whether the result is better.
• Anything where the diff is the conversation. If the value of the work is in going back and forth about it, batching it into one shot destroys the thing that made it work.
• Anything triggered by untrusted input without a hard allowlist. An @agent trigger on public issues is a prompt-injection surface with a repository token attached.

The dividing line is not task difficulty. It is whether "done" can be checked without you. A fiendishly complex codemod is a great CI job; a simple naming decision is a terrible one.

Where Backgrind fits

Straight answer: it does not, and CI is the one place that is true. Backgrind is a desktop overlay — it keeps your agents alive in a background daemon and pings you the moment one needs a decision, on top of whatever else is on your screen. A GitHub Actions runner has no screen and no you. Pitching an overlay into a pipeline would be nonsense.

What is real is the boundary between the two, and this post is mostly an argument about where it sits. CI is for the work that needs no judgment — bounded, specified, checkable, and safe to be wrong about because a human reviews the PR. Everything on the other side of that line is work where the agent will hit a question, and the entire value is in that question reaching you quickly instead of being auto-denied and silently reported as a success. That is the half Backgrind is for: policies handle what you can decide in advance, an ambient prompt handles what you cannot, and you can answer from your phone when the ping catches you away from the desk. Try the live demo if you want the feel of it.

Frequently asked questions