How does inconsistent agent behavior affect user understanding?

The paper's second overarching communication challenge, X2, asks "How should the agent convey consistent behavior?" It identifies two distinct sources of inconsistency. One is "the agent's behavior's inherent stochasticity," arising from "the probabilistic nature of the underlying foundation model's outputs" and from "the complex interaction patterns that emerge during task execution." The other is structural: "even with deterministic foundation model outputs (e.g., a model with temperature set to 0)," an agent's "orchestration logic — which governs action selection, inter-agent delegation, and task completion criteria — can produce different sequences of actions across runs as the environment changes in response to agent actions." The worked example is a visualization agent given the same high-level request repeatedly: it "might choose different libraries or tools to plot the visualization," sometimes fetching live data and other times "generate that data from its memory," and could "unpredictably apply different visual styles," which the paper calls "especially problematic if the user is unaware."

The paper does not treat variation itself as the problem — it notes stochastic behavior "may be desirable in some cases," such as generating multiple options to choose from — but argues that undisclosed inconsistency specifically "can hinder the user's ability to create an accurate mental model" of the agent. It also flags a second source of perceived inconsistency that is not the agent's fault: cases "when the user's mental model of the world does not match up with the information the agent is acting on."

This is a companion challenge to How can users verify what an AI agent actually plans to do? from the same paper, and the two compound rather than substitute for each other: verifying a plan at one moment does not help if the same request produces different agent behavior on the next invocation, silently and without explanation. It also complicates Does granting agents more autonomy undermine human oversight?: that paper locates the erosion of oversight in the overseer's own capacities being worn down by use; this excerpt locates a separate obstacle upstream of that, in the agent's behavior itself failing to hold still long enough for a mental model to form, independent of how skilled or attentive the overseer is.

The excerpt supports this with one illustrative scenario (the visualization agent), not measured data on how often orchestration-driven variation occurs across runs or how much it affects user trust; it names the problem and gestures at solutions existing elsewhere in the paper without stating them here. The implication it supports at that strength is a design requirement — that disclosing an agent's current plan or output is not sufficient for an accurate mental model unless the agent's behavior is also consistent across repeated invocations of the same request, or its variation is made visible to the user — not a claim about how inconsistent any specific deployed agent actually is.