Can you trust an AI agent if you never see it 'think' — or is that peek inside mostly theater anyway?

This explores whether people can build a reliable sense of how an AI agent behaves without seeing its reasoning, and whether seeing it would help much. The corpus has no study that tests this head-on. It does have something more surprising: the 'visible reasoning' people hope to inspect may not be a good window in the first place. Reasoning traces work more like persuasive performances than faithful records of computation. Models trained on logically invalid or corrupted traces perform nearly as well as models trained on correct ones, so the words on screen may not be what produces the answer Do reasoning traces show how models actually think?. Other work shows models can scale up their reasoning entirely inside hidden internal states, without producing any thinking text, which suggests that written-out reasoning comes from how models are trained rather than from anything reasoning requires Can models reason without generating visible thinking tokens?. If that's right, a user reading reasoning traces may end up with a confident but wrong mental model.

So what do users actually rely on? Research on how people mentally model dialogue agents finds the picture is dominated by perceived competence (about half the variance), followed by how human-like the agent seems and how flexibly it communicates How do users mentally model dialogue agent partners?. People mostly judge agents by outcomes and social cues, not by inspecting process. That has a known weak spot: matching outcomes can hide different processes underneath. Groups of language models reproduce the same overall pattern as human groups, where discussion helps weaker members more than strong ones, but they get there through more conformity, earlier convergence, and less sharing of unique information Do language model groups mimic human group reasoning patterns?. A user who sees only results would reasonably assume the agent works the way people do, and would be wrong.

There are alternatives to reasoning traces as a window into an agent. One is to expose the plan instead of the thinking. Cocoa lets users see an agent's plan, assign individual steps to themselves or the agent, and revise the plan mid-execution, which gave users noticeably more control than a chat interface did Can users steer AI agents better by revising plans mid-execution?. Another is structured memory: agents that fold their history into labeled episodic, working, and tool memory leave behind something a person could inspect Can agents compress their own memory without losing critical details?. A more speculative route skips language altogether. Researchers have pulled shared and private 'thoughts' directly out of agents' hidden states and caught disagreements before they showed up in any text Can agents share thoughts directly without using language?.

One more twist: the mental-modeling problem runs both ways. LLMs themselves fall back on surface strategies rather than genuinely tracking what another person believes, and hybrid systems that force explicit belief tracking do better Do large language models genuinely simulate mental states?. The same lesson may apply to users. Accurate mental models of agents probably come less from reading their thoughts and more from explicit, checkable structures such as plans, memory, and delegated steps that both sides can point to.