What is coding agent observability in 2026? From tokens to tickets

Open this month's usage report for your engineering org. On Claude Team and Enterprise plans it is a CSV (spreadsheet) with one row per user per model: requests, tokens, estimated spend. Cursor's admin page shows agent edits and "AI share of committed code". ChatGPT Enterprise shows credit spend by user, product and model, Codex included.

All of them answer "who used how much." None answers what the CFO (chief financial officer) asked: what did we get for it? Which tickets the tokens closed. Which pull requests took four human corrections before merge. Why one engineer burns three times the tokens of the person next to them on the same kind of work, and whether that is skill, a bloated CLAUDE.md, or a subagent loop nobody noticed.

That gap between the bill and the output is what coding agent observability closes. This post defines it, lists what to record, names the metrics worth reporting upward, and is honest about what visibility can and cannot prove.

A five-person team with no numbers

In July we talked to a team of four engineers and one product manager. Everyone used a coding agent. The CTO was clearly better at it than the rest and nobody could say why, because there was nothing to look at: no per-session record, no way to see which habits and which repo conventions produced clean pull requests and which produced rework. Leadership wanted the answer for real decisions, like whether to hire.

Five people. If a five-person team cannot see where the tokens go, a fifty-person org with three agent vendors and a usage-based plan certainly cannot.

Symptoms

Two or more of these and the rest of this post applies to you.

• The monthly AI coding invoice moved by more than 30% and nobody can name the tickets, repos or people behind the change.

• Two engineers close similar tickets and one uses several times the tokens, with no way to tell retries from a never-cleared context from a subagent fan-out.

• Agent-written pull requests get merged, then rewritten. "Vibe coding" is what your senior engineers call it in review, and not kindly.

• Every engineer keeps a personal CLAUDE.md or AGENTS.md, and the team's real conventions live in nobody's file.

• Finance asks for the return on the AI spend and engineering answers with "lines of code accepted."

The definition

Coding agent observability is recording three things about agents like Claude Code, Cursor and Codex, and joining them: what the agent did (traces, sessions, tool calls, subagents, the rules and skills it loaded), what it cost (tokens, cache hits and misses, dollars or credits per session), and what it produced (the ticket, the pull request, the CI result, the rework before merge). Joined, they let a person answer "where did the money go and what did we get" for a ticket, a team or a quarter. Separate, they are logs.

Call it monitoring, performance tracking or AI analytics; the joining is the whole definition. A trace with no cost is a debugging aid. A cost with no output is an invoice. An output with no trace is a leaderboard.

Why this is not LLM observability

Langfuse, Braintrust and Datadog LLM Observability are good products for agents your company builds: they record traces and spans, score outputs, and track cost per span for a loop you wrote. Coding agents break three of their assumptions. You do not own the harness: the loop, system prompt and compaction logic belong to Anthropic, OpenAI or Cursor, and you shape it only through CLAUDE.md, AGENTS.md, skills, hooks and MCP (Model Context Protocol, the standard for plugging tools into an agent) servers. The output lives somewhere else: a pull request on GitHub and a ticket in Linear, which now runs coding sessions through Claude Code or Codex straight from an issue. And billing changed shape: Claude Team and Enterprise meter usage credits past the seat allowance, ChatGPT Enterprise added credit analytics and per-team limits in June 2026, and Cursor moved to credits. On a seat plan the question was "are we using it enough." On a usage-based plan it is "was this session worth what it cost," and that needs session-level data.

What to record: four layers

Ordered from cheapest to collect to most valuable to report.

1. Execution: traces, sessions, tool calls, concurrency

A session is one conversation between a person and an agent, from first prompt to /clear. A trace is the ordered record inside it: each model turn, each tool call (read a file, run a command, edit, search), each subagent spawned with its own context window, each human message. All three tools keep this on the engineer's machine; Claude Code can also export it through OpenTelemetry (an open standard for shipping metrics and logs) as events like claude_code.tool_result and claude_code.api_request, with prompt text redacted by default.

The volume surprises people. In an internal replay across 45 of our own sessions, the agents made 6,910 tool calls, about 154 per session. That is where tokens go: not the prompt you typed, but the file reads, test runs and retries the agent chose for you.

Agent concurrency is the multiplier. Subagents, agent teams and any multi-agent setup each open a fresh context window; Anthropic's docs put agent teams at roughly 7× the tokens of a normal session when teammates run in plan mode. A trace that hides fan-out shows a "$40 session for a small ticket" with no explanation.

2. Harness: CLAUDE.md, AGENTS.md, skills, MCP, memory

This is the layer LLM observability tools never see, and where most avoidable spend hides.

Everything loaded at session start is paid for on every turn: CLAUDE.md and AGENTS.md (the instruction files agents read from your repo), skills, MCP tool definitions and whatever memory the agent carries. Anthropic's guidance is to keep CLAUDE.md under 200 lines and move workflow instructions into skills that load on demand. We covered the failure mode in Too many CLAUDE.md and skill files and the budget in Minimum viable context.

The observability question is drift detection: does the harness still match reality? One CLAUDE.md per engineer, each edited on a different Tuesday, is the drift. The fix your senior engineer worked out never reaches anyone else's agent, and nobody finds out until review. Record which rule files, skills and MCP servers each session loaded and you can see which rules are consulted and which sessions ran with none. Claude Code's /usage already attributes recent usage to skills, subagents, plugins and individual MCP servers per machine; that is the right shape, it just needs collecting across the team.

Governance lives here. "Never force-push" or "run the migration check before touching schema.sql" is only governance if you can show, per session, whether it fired. What is agent procedural memory covers the gap between a rule an agent reads and a rule it follows.

3. Money: tokens, cache, budget

Tokens are the unit; dollars or credits are the report. Three things matter beyond the total.

Cache hit rate. Coding agents resend the whole conversation every turn, and prompt caching is what makes that affordable. At Claude Opus 5 list price, input is $5 per million tokens and a cache hit is $0.50. A session that resends a 150k-token context 40 times moves 6 million tokens: about $3 warm, about $30 if the cache is cold each time. Same work, ten times the bill. (Illustrative; contracted rates differ.) A session that idled past the cache lifetime, or whose tool definitions changed mid-run, is a cost event worth seeing.

Per-session cost, not per-user. Per-user is what the vendor CSV gives you. Per-session lets you compare like with like: two bug fixes in the same repo, one at $4 and one at $38.

Budget control. Every vendor now offers spend limits: Claude at org, group and member; ChatGPT Enterprise at workspace, team and individual. A limit is a fuse, not a plan. The FinOps (cloud financial operations) job is to forecast from run rate and see which team is over before the fuse blows. Anthropic's published enterprise average is $150 to $250 per developer per month; for 40 engineers that is $6,000 to $10,000 a month before anyone has looked at where it went.

4. Outcomes: tickets, pull requests, CI, rework

This is what turns the other three into an answer for the CEO and CFO.

The unit of account is the ticket (or the PR, if that is how you count work). Every session, subagent and retry that touched it is attributed to it. Then:

• Token-to-ticket ratio, reported as cost per accepted ticket: all tokens on the ticket, including failed attempts, retries and subagents, divided by one merged result, split by work type (feature, bug, chore) so you stop comparing a migration with a typo fix.

• Human messages per PR: how many times a person had to redirect the agent before the change was mergeable. The cheapest proxy for supervision cost.

• Rework rate: share of agent-written lines changed between first draft and merge, kept separate from ordinary post-merge bug fixing.

• CI (continuous integration) outcome per session: did the agent's change pass on the first run.

Evaluation and benchmarks. Public benchmarks tell you which model is strong in general and nothing about your codebase. The only evaluation that matters for spend decisions runs on your own tickets: same work type, same repo, before and after a change to the model, the harness or the rules.

What a coding agent observability dashboard should show, by role

The same data, four views. Each role opens the dashboard with a different question and a different decision behind it. A dashboard that shows everyone everything gets ignored by everyone.

Tech lead: this week's sessions

The tech lead is looking for the session that went wrong before it becomes a pull request that goes wrong. The view is a list of the team's sessions for the week, sorted by cost, with each row showing tokens, turns, tool calls, subagents spawned, human messages, cache hit rate and the ticket it touched. From the list they need to open any session and read it turn by turn.

Two filters matter most: "sessions over 3× the team's median cost" and "sessions that loaded no team rules." The first catches loops and fan-out. The second catches drift. Decision: which two sessions to walk through at the next team meeting, and whether a CLAUDE.md or skill needs fixing today.

Engineering manager: cost per ticket and supervision, by team

The manager wants the trend, not the session. The view is a month of cost per accepted ticket split by work type (feature, bug, chore), human messages per PR, rework rate, and the same three numbers for the previous month, with a similar-work comparison across teams rather than a raw ranking. A second panel shows who is spending on what: sessions and cost by engineer, but only within a work type, so a migration is never compared with a typo fix.

Decision: where to coach, which team's harness to copy, and what to say when someone asks whether the agent budget should go up.

CTO: quality, governance and concentration

The CTO's question is whether the organization can keep doing this at scale. The view has three panels. Quality: rework rate and CI first-pass rate for agent-written changes against human-written ones, by repo. Governance: harness coverage, meaning the share of sessions that loaded the current team rules and skills, plus a log of which rules fired and where they were bypassed. Concentration: what share of total tokens the top 10% of sessions consume, and whether that share is growing. When a few sessions eat most of the budget, the fix is usually one workflow, not a policy.

Decision: which vendor and model mix to standardize on, which rules to make mandatory, and whether the token bill is a scaling problem or a handful of bad habits.

CFO and FinOps: spend against budget, forecast, and unit cost

Finance wants three numbers per team and one for the company: spend to date against the budget, forecast to month end at the current run rate, and cost per accepted ticket as the unit cost that lets them compare months. Alongside: the split of spend across new features, rework, tests and reviews, and exploration, and the share of spend that is usage credits above the seat allowance, since that is the line that moves on the invoice.

Decision: whether to approve the next budget request, where to set spend limits, and what to put in the board deck. The board gets three numbers, not twelve.

What not to do

Do not report lines of code accepted as productivity. It counts what the agent typed and the person did not reject in the moment; Claude Code's own docs say it "does not track subsequent deletions."

Do not rank engineers on a token leaderboard. It measures enthusiasm and context bloat. To find the people worth learning from, sort by cost per accepted ticket within a work type, then read their sessions.

Do not treat agent time as output. Agent hours divided by human hours is descriptive. A session that ran three hours without a human message may have been productive or may have been looping. Pair it with accepted output before quoting it.

Do not compare teams without comparing work. A platform team doing migrations will always cost more per PR than a growth team shipping copy changes.

Do not trust attribution blindly. Claude Code's PR attribution matches sessions to merged PRs from 21 days before to 2 days after merge and drops any line a human rewrote by more than 20%. Sensible, and still a heuristic. Know the rules of whatever attribution you rely on.

Buy, build, or both

What you already have. Claude Code, Cursor and ChatGPT Enterprise each report usage and spend per user, and Claude Code and Cursor add PR and committed-code counts. All three stop at the user; none joins a session to a ticket or sees the other vendors. Claude Code can also stream tokens, cost, tool results and API events over OpenTelemetry to any collector, which is how you get raw events into Datadog next to production telemetry. Langfuse and Braintrust will store coding-agent traces if you pipe them in, but ticket, PR, harness file and budget are not concepts they model. A newer group of session-level tools, XTrace among them, captures the Claude Code, Cursor and Codex session itself and links it to the pull request and ticket.

What building it costs. Turn on OpenTelemetry export on every machine and run a collector. Decide whether to set OTEL_LOG_USER_PROMPTS=1, because without it prompts are redacted and you cannot tell a feature session from a bug session. Discover that Cursor and Codex do not speak the same export, so you are parsing local session files in three formats that change with each release. Then write the join: sessions carry a repo path and maybe a branch, tickets carry an identifier, PRs carry a branch; handle the engineer who works two tickets in one session and the subagent whose output landed in a different PR; decide what "rework" means in your diff tool. Two to three engineer-weeks for a first version, and a recurring tax with every vendor release.

What visibility does not prove

Visibility explains how a team found and fixed waste. It is not, by itself, ROI (return on investment). Seeing that one repo's sessions cost twice the org average is a finding; the return exists only after someone changes the harness and next month's cost per ticket comes down while rework stays flat. Missing baseline data is unknown, not zero: with no history, the first report is the baseline and the second is the first result. Anyone showing you a return number in week one, us included, should be asked for the before.

Start with one month and one team

You do not need a platform to start. Pick the team with the highest agent spend, pull one month of their sessions, and compute four numbers by hand: cost per accepted ticket by work type, human messages per PR, rework rate, and the share of sessions that loaded the team's current rules. That is a baseline, and it usually surfaces one workflow worth fixing on its own.

If you would rather not build the join yourself, this is the problem XTrace Sessions was built for: it captures Claude Code, Cursor and Codex sessions and produces the four role views above across teams. You can talk to us with a month of your usage report in hand.