Frontier model silently leaked production secrets, despite instructions not to

At a glance

The assumption that didn’t hold

It’s easy to assume today’s frontier models are too capable to leak a production secret. Every incident in this case study involved Claude Opus 5.5, one of the most capable coding models available. It knows what a credential looks like, and it knows not to expose one.

The agents did know. Before each exposure, they took a precaution. Often the only boundary was an instruction to stay read-only, and printing a secret still counts as a read. Other times, the agent wrote its own filter to hide the values. The precaution failed, the secret landed in the session transcript, and in nine of eleven cases the agent carried on without a word.

That is the finding that matters to anyone accountable for AI agents in their company. Even frontier models break security policy at runtime, while trying to follow it, and they don’t reliably tell you when they do.

What happened

The client is a software company whose engineers have built with AI coding agents for years. Agents are part of how they ship every day. XTrace recorded 699 of their agent sessions. Nobody asked an agent to handle credentials. Every exposure happened during routine work, usually debugging. Each exposed value went to the model provider with the agent’s next request and was saved in the session transcript on the engineer’s machine.

One tool call, up close

A redaction filter that redacts nothing. To check a configuration file without exposing it, the agent piped it through a filter meant to blank out every value. Simplified, with secret values removed:

# Meant to show the keys and hide the values.
$ sed -E 's/(:)\s.*$/\1/' config/staging.yml

# macOS sed doesn't support \s, so nothing matches
# and the whole file prints. Even with GNU sed,
# the multi-line key below would still print:
auth:
  client_secret: <printed in full>
  private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    <printed in

# Meant to show the keys and hide the values.
$ sed -E 's/(:)\s.*$/\1/' config/staging.yml

# macOS sed doesn't support \s, so nothing matches
# and the whole file prints. Even with GNU sed,
# the multi-line key below would still print:
auth:
  client_secret: <printed in full>
  private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    <printed in

# Meant to show the keys and hide the values.
$ sed -E 's/(:)\s.*$/\1/' config/staging.yml

# macOS sed doesn't support \s, so nothing matches
# and the whole file prints. Even with GNU sed,
# the multi-line key below would still print:
auth:
  client_secret: <printed in full>
  private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    <printed in

The agent’s homemade redaction was broken twice over. Claude writes shell the Linux way by habit, and on a Mac that pattern matches nothing. Even where it works, it only blanks a value on the same line as its key, so a multi-line private key prints anyway. Private keys and client secrets printed in full, and the agent moved on as if the values were hidden. Claude Code folds command output out of view, and this ran during a long stretch with no engineer watching. Once, it happened inside a helper agent, whose work engineers see even less often. Nothing about this is specific to one company: it can happen anywhere an agent’s habits don’t match the machine it runs on.

Why existing controls saw nothing wrong

Every exposure was built from steps that look legitimate to the systems companies rely on today. An experienced engineer, with valid credentials, used an approved AI tool to run an allowed command. No single event was the problem. The agent was supposed to keep these values hidden, didn’t, and then said nothing.

By the time a problem like this surfaces in a log review, the secret is already stored in the session transcript and logs, and everywhere those are retained.

Silence is the real risk

The agents weren’t hiding anything on purpose. They treated a step as safe because of what it was meant to do: the command was read-only and the filter was meant to redact, so they moved on without checking the output. That habit matters because of how experienced teams actually work with agents.

Engineers have become approvers more than authors. When most work happens between check-ins, the agent’s own report is the main source of oversight, and this case study shows it isn’t a reliable one.

How the XTrace plugin closes the gap

The XTrace plugin is built for this category of risk. Its rules aren’t advice the agent weighs against its task. Each one is a hard stop, set by the security team and checked outside the model before the action runs. The agent can’t switch it off, retry its way past it or hand the job to a helper agent to get around it. When a rule blocks an action, the action doesn’t happen, and a final block stays in place until a person lifts it.

Every major coding agent has its own way to enforce rules, configured separately, and each update can open a gap. XTrace keeps one rulebook for all of them. It watches how your agents actually work, spots risky patterns, and proposes rules your security team approves, each tested against your own session history first so you know what it will block before it goes live. With the client’s security team, XTrace set up rules for each risk this case study surfaced: secret values in command output, secret files and keys, production changes and CI workflows. Your team can add its own rules at any time.

Alongside the rules, the client rotated the exposed secrets, moved production secrets into a secrets manager and limited the production deploy role to the production environment.

A secrets manager shrinks the surface but can’t close it. An agent that needs to read production logs needs a key, and a key it can use is a key it can print. The rule has to sit where the agent acts.

The plugin isn’t a sandbox, and it doesn’t replace the controls you already run. It works alongside them: a secrets manager to keep secrets off laptops, the agent’s own sandbox to limit what it can touch, and review on changes that reach production. Its job is the moment of action, which those controls can’t see.

What a gateway catches, and what it can’t

A gateway on model traffic can catch a secret on its way to the model provider, if it inspects content. It can’t stop the file read, the production write or the CI change, because those happen on the engineer’s machine, at the moment an agent acts. That is where the check has to be.

Security and platform leads write the policy once. Every coding agent in the company follows it. See how XTrace enforces policy in coding agents.

Questions to ask about your own agents

• If an agent printed a production secret today, how would you find out, and how long would it take?

• Which of your security rules for agents are enforced, and which are only instructions in a prompt?

• Can you see what helper agents do, or only the main agent’s summary?

• Can you trace each agent action to a person, an agent and a policy?

About this case study

XTrace recorded 699 Claude Code sessions and 2,793 helper agents with the client. Every security incident involved Claude Opus 5.5. Incidents were classified against the OWASP Top 10 for LLM Applications and MITRE ATLAS, and exposed secrets were confirmed with the Gitleaks scanner. Session data was encrypted in transit and at rest, secret values were redacted from everything in this report, and the client rotated every exposed secret. Findings describe this client, not agent behavior in general.

Find out what your agents do between check-ins. Request an XTrace demo.