Can execution traces ground honest explanations of agent behavior?
Multi-step AI agents leave execution traces behind, but can these raw records be converted into reliable, auditable explanations that expose unsupported claims? The question matters because human oversight of autonomous agents depends on trustworthy process-level transparency.
The paper argues that agents acting through external tools and sequential decisions "with limited human oversight" create a need for "reliable and auditable explanations of what an agent did and why," and that traditional Explainable AI methods "fall short of providing the process-level transparency" such systems require. Its answer is a post-hoc framework that takes a lengthy execution trace and produces two things: a structured report and a "faithful natural-language explanation explicitly grounded in its observable behavior." According to the abstract, human and automated evaluations across multiple benchmarks and architectures show high-quality, trace-faithful explanations that reliably identify unsupported claims, unjustified actions, and evidence gaps, and that outperform naive LLM-generated explanations.
The mechanism is the choice of evidence. The framework "relies solely on execution traces," which is why the authors say it applies across agent architectures, environments, and tasks: the trace is the one artifact every tool-using agent leaves behind, so the explainer does not need access to the agent's internals or design. The structured report sits between the raw trace and the prose, so the natural-language explanation can be checked against something. That is the contrast with a naive LLM-generated explanation, which can sound coherent without being tied to what the agent actually did. The introduction motivates the design with failure modes it attributes to prior studies, namely error cascades, responsibility gaps, flawed execution monitoring, and multi-step error propagation that "obscures the origins of system behavior and outcomes."
This adds an explanation-side answer to notes that treat the execution record as evidence about agent behavior. Do automated benchmarks hide what frontier AI systems can really do? relies on small-sample qualitative log analysis to assess capability; this paper works on the same kind of record but aims at a report a person can audit, and it does so per run rather than per evaluation study. The framework's targets, unsupported claims and unjustified actions, are the kind of failure described in Does completion training push agents to overfill forms unnecessarily?, although the excerpt never connects the two. It also speaks to the transparency and accountability gap that Should AI systems stay collaborative rather than fully autonomous? names as a reason to keep humans in the loop: a trace-grounded report is one way to give that human something to check.
The excerpt is only the abstract and one introduction passage, so a good deal is left open. It does not describe how the structured report is built, how unsupported claims or evidence gaps are detected, which benchmarks and architectures were used, how the human evaluation was run, or how large the advantage over naive explanations is. It also does not say whether an explanation confined to observable behavior can recover the agent's internal reasons for an action, which matters because the stated goal covers "why" as well as "what." Until those details are available, the safe reading is narrow: grounding explanations in the trace is a workable way to keep them faithful and to surface gaps in the evidence, and the paper reports that it beats unstructured LLM explanations. Whether it changes how well people oversee agents is not something the excerpt shows.
Inquiring lines that read this note 27
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do users build accurate mental models when agent reasoning is partially hidden?- Can explanations grounded in observable behavior recover an agent's internal reasons for acting?
- Can execution traces reveal unsupported claims in AI agent behavior?
- Can showing alternative paths an agent rejected improve trace comprehension?
- What should an AI interface surface to disclose an agent's proposed actions?
- What does it mean for users to verify an agent's plans before execution?
- Do trace interfaces actually help users find and catch agent errors?
- What surfaces should expose an agent's plan before execution begins?
- Can group members audit or understand how an agent processes their data?
- Can an auditor verify environment state without trusting the executor's self-report?
- Can execution traces expose methodological shortcuts that output checks would miss?
- Do execution trace reports change how developers review agent output?
- How do action receipts differ when an agent executes without human review?
- Does removing approval steps before execution change what a receipt must report?
- Can users distinguish between automation errors and unauthorized agent actions?
- Why do auditors struggle to spot errors in agent work after execution?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do automated benchmarks hide what frontier AI systems can really do?
Benchmarks optimize for auto-gradable, short, cheap tasks. But real AI capability emerges in long-horizon, messy, open-ended work. How much capability are we missing—or wrongly inflating—by relying on benchmark scores alone?
also treats agent logs as primary evidence; this paper turns a trace into an auditable per-run explanation rather than an evaluation
-
Does completion training push agents to overfill forms unnecessarily?
Explores whether agents trained to complete tasks end up filling optional fields they shouldn't touch. This matters because it creates privacy risks from over-helpfulness rather than malice.
describes over-claiming failures of the sort the framework's unsupported-claim detection targets; the excerpt does not link them
-
Should AI systems stay collaborative rather than fully autonomous?
Explores whether keeping humans in the loop with AI agents is more reliable than pursuing full autonomy. Investigates whether collaboration solves problems that autonomous systems structurally cannot.
names transparency as an autonomous-agent failure; this paper proposes a trace-based mechanism for it
-
Can a correct outcome hide protocol violations in multi-agent systems?
When agents reach the right verdict without following required steps, how can we tell if they complied or cut corners? Outcome-level checks alone may miss the difference.
Evidence for: a verdict matching ground truth hid agents skipping the log check, so outcome-level checks miss what trace-grounded, process-level explanation would expose
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Explaining AI Agents Through Execution Traces
- Overseeing Agents Without Constant Oversight: Challenges and Opportunities
- A Black Box for Agentic Processes: Blockchain-Anchored Evidence for AI Agent Communication, Human Oversight, and GRC Audits
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
- Demystifying Agent Skills: Why They Work-Until They Don't
- Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents
- BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure
- Agentic Code Reasoning
Original note title
post-hoc explanations of AI agents should be grounded in the execution trace — a structured report exposes unsupported claims and evidence gaps