If an AI shows you its 'thinking,' does that actually help you predict what it'll do next — or just feel reassuring?
What role does visible agent reasoning play in user mental models?
This explores whether letting users see how an agent is thinking (its reasoning steps, plans or working memory) helps them build an accurate picture of what the agent is and what it will do next.
This explores whether showing users an agent's reasoning helps them understand what the agent is actually doing and predict what it will do next. The corpus doesn't have a study that directly tests reasoning displays on users. Read together, though, several notes suggest an answer: visibility matters a lot, but the reasoning text a model prints is not necessarily the thing users need to see.
Start with the user's side. People size up dialogue agents mainly on perceived competence, with human-likeness and communicative flexibility well behind it How do users mentally model dialogue agent partners?. Users are therefore watching for signals of whether the agent knows what it's doing. Agents are hard to read, though: model randomness and orchestration that reacts to the environment make them behave differently from run to run. Users can't form a reliable mental model unless that variation is either reduced or made visible How does inconsistent agent behavior affect user understanding?. That is the strongest argument for visible reasoning. It turns 'the agent did something different this time' into 'the agent did something different because it saw X.'
There's a catch. Models can scale their reasoning in hidden internal states without writing out any thinking tokens, which suggests the step-by-step text may be partly a byproduct of training rather than the reasoning itself Can models reason without generating visible thinking tokens?. When the reasoning is written out, it can look messy. Reasoning models wander through invalid paths and drop promising ones too early Why do reasoning models abandon promising solution paths?. A user reading that trace might decide the model is either more careful or more confused than its final answer suggests. Either way, the visible narrative and the real process can drift apart, and the user's mental model is built on the narrative.
The more promising view in the corpus is that what users see should be the agent's working materials, not its inner monologue. Code works well here because it is executable, inspectable and stateful at once, so you can check what the agent actually did instead of what it says it did Can code serve as the operational substrate for agent reasoning?. JarvisHub places prompts, references, versions and feedback as typed nodes on a canvas that both the human and the agent use, replacing hidden chat state with something both can point to Can a shared canvas serve both human and agent memory?. This matches the finding that agent reliability comes from moving memory, skills and protocols out of the model and into an external structure called a harness Where does agent reliability actually come from?. The same move that makes an agent reliable also makes it readable.
The takeaway: the most useful thing to show a user may not be what the agent is thinking. It may be what the agent is working with and what it changed. Reasoning text can mislead, while external state can be checked, and a mental model built on checkable state holds up better when the agent's behavior varies.
Sources 7 notes
The Partner Modelling Questionnaire reveals that perceived competence dominates user impressions (49% of variance), followed by human-likeness (32%) and communicative flexibility (19%). This three-factor structure reflects how people evaluate dialogue partners against both functional and social standards.
Stochasticity in foundation models and environment-reactive orchestration logic cause agents to behave unpredictably across runs. Users cannot develop reliable mental models unless variation is either minimized or made visible.
Multiple architectures—depth-recurrent models, Heima, and Coconut—demonstrate that test-time compute scales through hidden state iteration rather than token generation. This suggests verbalization is a training artifact, not a reasoning requirement.
Reasoning LLMs exhibit two reinforcing failures: wandering (invalid exploration) and underthinking (premature path-switching). Decoding-level interventions like thought-switching penalties improve accuracy without fine-tuning, suggesting viable solutions exist but are abandoned prematurely.
Research shows code uniquely enables agent reasoning, action, and verification by being simultaneously executable, inspectable, and stateful. This unified code-centered loop improves reasoning and verification together compared to natural-language or prose-based approaches.
Show all 7 sources
JarvisHub proposes that placing prompts, references, versions, and feedback as typed canvas nodes visible to both users and agents—rather than hiding agent memory in chat or transient state—enables local updates, artifact reuse, and unfinished work continuation without process opacity.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Code as Agent Harness
- Towards a Science of AI Agent Reliability
- A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments
- Challenges in Human-Agent Communication
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Reasoning LLMs are Wandering Solution Explorers
- Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Models
- Agentic Code Reasoning