Users mostly love what AI agents produce — but lose track once they look away mid-task; why does that gap keep happening?

This explores what people need to understand about AI agents, and what they need to be able to see, to supervise them well. The corpus suggests the hard part is not judging whether an agent's output is good. It is understanding what happens between the instruction and the result. A study of 73,093 Reddit posts about the agent OpenClaw found that users' values were mostly met when they described what the agent delivered, but went unmet in every user group when they described supervising it Where do user values break down in agent supervision?. People broadly like what agents produce. Things fall apart in the middle, where users set the agent up, lose sight of it while it works, and then try to make sense of what it did.

One reason is that people bring the wrong default picture to agents. Research on how users model dialogue partners found that impressions mostly come down to perceived competence (49% of the variation), then human-likeness (32%), then communicative flexibility (19%) How do users mentally model dialogue agent partners?. That picture works for judging a conversation partner. It doesn't help much with overseeing a worker that takes actions on its own. A more useful model asks a different question: not "how smart is it?" but "what is it allowed to decide without me?" Work on autonomy alignment treats this as separate from getting good outputs. It splits the question into three layers: when the model chooses to check in with you, what the surrounding system actually enforces, and how the interface shows you what you've handed over How should AI agents balance autonomy with user oversight?. A user who only thinks about the model misses the other two layers.

A second mental-model shift: an agent's reliability often doesn't come from the model at all. Research on agent design argues that dependable agents move memory, skills, and interaction protocols out of the model and into a surrounding "harness" Where does agent reliability actually come from?. Related work describes code as a medium you can run, inspect, and check, so an agent's reasoning leaves a trail you can examine afterward Can code serve as the operational substrate for agent reasoning?. For supervisors, this means oversight works best when you watch the visible structure (the plan, the logs, the state) rather than trying to read the agent's mind. The Cocoa system puts this into practice. Users see the plan, assign individual steps to themselves or to the agent, and revise steps while the work is running. In a lab study, this gave people more control than a chat interface without making the tool harder to use Can users steer AI agents better by revising plans mid-execution?. Cognitive science work on AI "thought partners" makes the same point from another direction. Working together well depends on legibility and a shared model of the world, so each side can predict the other What makes an AI a true thought partner, not just a tool?.

The corpus also warns that individual users may not be able to carry this load alone. Agency law and economics name three old problems in any principal-agent relationship: the agent knows more than you do, has room to use its own judgment, and may not be loyal to your interests. Their analysis argues that the usual fixes (incentives, monitoring, enforcement) break down when the agent's decisions are opaque and happen fast and at scale Can traditional governance tools actually control AI agents?. Deloitte's answer is to stop treating oversight as purely a user skill. It proposes placing humans at deliberately designed handoff points, backed by digital identity systems, cryptographic receipts, and permanent action logs How should organizations supervise autonomous AI agents?. Other researchers argue the frame itself is too narrow, because agents acting in the world affect people other than the one giving instructions Should AI agents serve one user or balance many interests?.

The takeaway you might not have expected: the most useful mental model for supervising an agent is closer to managing a delegated workflow than to talking with a capable assistant. Know which decisions you've handed over, look at the plan and the trail of actions rather than the personality, and expect supervision, not output quality, to be where your expectations and the agent's behavior come apart. The corpus has less on how to teach users these models directly. Most of the work here describes what the system and interface should show, not how to train people to think differently.