SYNTHESIS NOTE
Topics›Agents›this note

How can users verify what an AI agent actually plans to do?

Stating goals to an AI agent isn't enough for mutual understanding. What mechanisms would let users check both the agent's proposed plan and whether it truly grasps the user's goals and constraints?

Synthesis note · 2026-10-19 · sourced from Agents

Writing from "a communication grounding perspective," the paper identifies twelve communication challenges that tool-using, multi-step AI agents pose to users, organized into overarching challenges, information flow from the user to the agent, and information flow from the agent to the user. The excerpt elaborates the first overarching challenge, X1: "How should the agent help the user verify its behavior?" Its answer is that "it is not enough... for a user to state their goals and preferences if these goals and preferences are misunderstood or ignored by the agent" — the user must also be able to "verify their understanding of the agent's plans and actions as well as the agent's understanding of their own goals, preferences, requirements, and feedback." The illustrative case is an agent fixing a GitHub bug: it "might fail to understand the issue and its requirements," "might develop a plan that implements the fix... but neglects to update the test cases," "might propose a fix that is 'unsafe'... or convoluted and time-consuming to review," or "might propose a change that has side effects such as removing unrelated functionality."

The reasoning draws on Clark and Brennan's grounding theory: common ground is "the collection of mutual knowledge, mutual beliefs, and mutual assumptions" that participants build by coordinating "both the content and process" of a conversation, and grounding is the process of reaching that shared belief. Brennan's extension to human-computer interaction notes that "people need to be able to seek evidence that they have been understood and to provide evidence about their own intentions." The paper argues agentic workflows make this harder because tasks are "decomposed and carried out over multiple steps — sometimes by multiple interacting agents with different roles, privileges, and access to different information," so conveying information once is not sufficient; verification has to be built in as a mechanism, or the user cannot "steer agent behavior to avoid costly mistakes," build "an accurate mental model," or give useful feedback.

This names a channel that Does granting agents more autonomy undermine human oversight? treats as already broken: that paper argues current agent design erodes the situational awareness, judgment, and skill oversight needs, without specifying what a working channel would look like; this paper's X1 is one candidate requirement — a mechanism for the user to check both the agent's plan and the agent's grasp of the user's goals — prior to any erosion from extended use. It also gives theoretical grounding to the practitioner proposal in How should AI interfaces handle the shift from doing to supervising?: Nielsen's Orchestration Surface, which discloses "its proposed plan," exposes "provenance," seeks "consent," and issues "post-action receipts," is a concrete design answer to the verification requirement this paper derives from grounding theory rather than from UX practice.

The excerpt covers one of twelve challenges (and only the setup for a second, discussed separately); it offers the GitHub bug-fix scenario as an illustration, not as measured data on how often such failures occur, and states "possible solutions" exist elsewhere in the paper without including them here. It is a problem-framing contribution, not an evaluation of any particular verification mechanism, so it supports treating verifiability as a design requirement for agentic systems, not a claim about how well any current agent satisfies it.

Inquiring lines that read this note 3

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Can post-hoc execution traces provide sufficient accountability without real-time oversight? How does transparency of agent reasoning affect effective oversight?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
15 direct connections · 138 in 2-hop network ·dense cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

users must be able to verify an agent's plans and actions to establish common ground with it