If you can't see how an AI agent reasons, how do you figure out whether to trust it?

This explores how people build a working picture of what an AI agent is doing and why, when they can't see its reasoning and the parts they can see may be misleading. The short version from this corpus: users mostly build their mental models from patterns in the agent's behavior, not from its explanations. That makes consistency more important than transparency, and it means visible reasoning can hurt as much as it helps. One caveat: the collection has little direct research on how users form these models. Most of what follows comes from research on the agent side.

The most direct material is on consistency. Because foundation models are stochastic and orchestration logic reacts to the environment, the same agent can behave differently on identical tasks across runs. When that variation is hidden, users can't form an accurate mental model at all (How does inconsistent agent behavior affect user understanding?). The proposed fix isn't more explanation. Either reduce the variation, or make it visible so users learn that the agent varies instead of building a confident but wrong picture from a few lucky runs.

If visible reasoning traces were faithful, users would have an easy shortcut. They aren't faithful enough to count on. Reasoning traces work more like persuasive performances than records of the actual computation. Invalid logical steps perform nearly as well as valid ones, and corrupted traces still produce comparable gains (Do reasoning traces show how models actually think?). Some models skip verbal reasoning entirely and do the work in hidden internal states (Can models reason without generating visible thinking tokens?). So a user who reads the 'thinking' and forms a mental model from it may be modeling a story the agent tells, not the process that produced the answer. This connects to research on groups of LLMs: they reach the same outcomes as human groups through quite different processes, with more conformity and earlier convergence (Do language model groups mimic human group reasoning patterns?). Results that look familiar can hide unfamiliar mechanics.

The most interesting lateral answer is architectural. If the model's internal reasoning is opaque, move more of the agent's work into places people can inspect. Reliable agents push memory, skills, and interaction protocols out of the model and into a surrounding system layer (Where does agent reliability actually come from?). Code works as a medium that is executable, inspectable, and stateful all at once (Can code serve as the operational substrate for agent reasoning?). These changes were made for reliability, but they also give users something concrete to reason about: state they can read and steps they can check, not a black box with a narrative attached.

The takeaway you might not have expected: the route to a good mental model of an agent may run through system design, not explanation. Predictable behavior and inspectable artifacts give users firmer ground than any reasoning trace, because the trace is the part most likely to look convincing while being wrong.