Telling an AI to 'think step by step' doesn't actually help it follow what people believe and want mid-conversation.
Does chain-of-thought reasoning improve mental state tracking in dialogue?
This explores whether prompting a model to 'think step by step' actually helps it keep track of what speakers believe, want, and know across a conversation — or whether mental-state tracking needs something CoT can't supply.
This explores whether chain-of-thought (CoT) reasoning genuinely improves a model's ability to track what conversational partners believe and intend — and the corpus answer is a careful 'not by itself.' The catch is what CoT actually is. Several notes argue that chain-of-thought reproduces the *form* of reasoning through learned pattern-matching rather than performing real inference: structurally invalid prompts work as well as valid ones, format matters more than logical content, and performance degrades predictably the moment you step outside the training distribution What makes chain-of-thought reasoning fail in language models? Does chain-of-thought reasoning reveal genuine inference or pattern matching? What makes chain-of-thought reasoning actually work? Does chain-of-thought reasoning actually generalize beyond training data?. Mental-state tracking in open dialogue is precisely the kind of novel, shifting situation that exposes imitation — so generating more reasoning tokens doesn't reliably buy you better belief-tracking.
The theory-of-mind work makes this concrete. LLMs pass tidy structured benchmarks but default to surface-level strategies in open-ended scenarios, failing at genuine perspective-taking — and crucially, the fix that worked was architectural, not more prompting: hybrid Bayesian systems that *force* explicit belief tracking outperformed the LLM-alone approach Do large language models genuinely simulate mental states?. That points away from 'reason harder in free text' and toward 'give the model a structured slot where beliefs are represented and updated.'
The dialogue-specific note sharpens it further. Collaborative Rational Speech Acts (CRSA) tracks *both* speakers' beliefs across turns by grafting rate-distortion theory onto pragmatic reasoning, capturing the progression from partial to shared understanding in things like doctor-patient exchanges — and it's framed explicitly as the information-theoretic scaffolding that token-level LLM systems lack Can dialogue systems track both speakers' beliefs across turns?. So the lever that helps isn't chain-of-thought as a style of text; it's a dedicated mechanism for representing who-knows-what.
Where reasoning *does* help is when it stops being pure internal monologue and gets grounded. Interleaving reasoning with external feedback — querying a tool or environment between thoughts — prevents error propagation that pure CoT suffers from Can interleaving reasoning with real-world feedback prevent hallucination?. The analogy to dialogue is direct: a partner's next utterance is real-world feedback on whether your model of their mind was right, and a system that updates against that signal will track intent better than one spinning out longer unanchored chains.
The quiet bonus finding: more reasoning is often worse. Optimal CoT length follows an inverted-U, capable models prefer *shorter* chains, and minimal drafts match verbose ones at under 8% of the tokens — most of the removed text was style, not computation Why does chain of thought accuracy eventually decline with length? Can minimal reasoning chains match full explanations?. For belief-tracking the implication is liberating: the win comes from a structured place to store and revise beliefs and from grounding against what the other person actually says — not from making the model narrate more.
Sources 9 notes
Research shows CoT mirrors reasoning form without true logical abstraction. Format matters more than content, invalid prompts work as well as valid ones, and scaling reasoning creates instruction-following deficits.
CoT works by constraining models to reproduce familiar reasoning patterns from training, not by enabling novel symbolic reasoning. Performance degrades predictably under distribution shifts—the signature of imitation rather than capability emergence.
Research shows training format shapes reasoning strategy 7.5× more than domain, demo position swings accuracy 20%, and invalid CoT prompts work as well as valid ones. CoT is pattern-guided generation, not formal logic.
DataAlchemy experiments show CoT fails systematically under distributional shifts in task, length, and format. Models produce fluent but logically inconsistent reasoning — imitating reasoning form without valid underlying logic.
ChangeMyView and FANTOM benchmarks show LLMs fail at authentic perspective-taking in open-ended scenarios, despite succeeding on structured tasks. Hybrid Bayesian architectures that force explicit belief tracking outperform LLM-alone approaches, suggesting the gap is architectural rather than merely training-based.
Show all 9 sources
CRSA integrates rate-distortion theory with RSA to enable bidirectional belief tracking across dialogue turns. Demonstrated on referential games and doctor-patient dialogues, it captures progression from partial to shared understanding, providing the information-theoretic framework that token-level LLM systems lack.
ReAct demonstrates that alternating verbal reasoning with external tool queries (Wikipedia API, environment interaction) prevents error propagation by injecting real-world feedback at each step. On knowledge-intensive and interactive tasks, this approach outperforms pure chain-of-thought and reinforcement learning by 10-34% absolute accuracy.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Chain of Draft achieves equivalent accuracy to standard chain-of-thought on arithmetic, symbolic, and commonsense tasks while using only 7.6% of tokens. The 92.4% of removed tokens served style and documentation, not computation.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- Break the Chain: Large Language Models Can be Shortcut Reasoners
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Hierarchical Reasoning Model
- CoT is Not True Reasoning, It Is Just a Tight Constraint to Imitate: A Theory Perspective
- Measuring Faithfulness in Chain-of-Thought Reasoning
- Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
- What Makes Effective Supervision in Latent Chain-of-Thought? An Information-Theoretic Analysis