Can an AI reason its way into understanding what someone else believes or wants — or does that take something more?
Can LLMs simulate theory of mind through reasoning alone?
This explores whether LLMs can get to real perspective-taking (tracking what someone else believes, wants, or doesn't know) by thinking harder step by step, or whether something besides reasoning is needed.
This explores whether LLMs can model other minds just by reasoning more, without any extra structure. The corpus mostly says no, and one finding cuts against intuition: more reasoning can make it worse. On the Decrypto benchmark, which tests false belief, counterfactual thinking, and noticing when your own view has changed, reasoning-optimized models like o1 and Claude 3.7 Sonnet scored below older models, below humans, and even below simple word-embedding baselines Why do reasoning models fail at theory of mind tasks?. Training a model to solve math-style problems doesn't carry over to social understanding, and it may crowd it out.
Part of the reason is how LLM reasoning works in the first place. When the meaning of a task is stripped away and only the logic remains, performance collapses, even with the correct rules sitting in context Do large language models reason symbolically or semantically?. Models lean on familiar associations rather than manipulating abstract structure. That explains a common pattern in theory-of-mind results: LLMs pass the structured, textbook-style tests but fail in open-ended conversation. In those settings they fall back on surface cues instead of actually simulating what another person believes Do large language models genuinely simulate mental states?. The same gap shows up in persuasion. Models barely agree with humans about which arguments actually changed someone's mind. They weight topical overlap and credibility, while people respond more to novelty and assertiveness Do language models judge persuasion the way humans do?.
What does help is structure, not more reasoning. Hybrid systems that force the model to write down and update explicit beliefs, Bayesian-style, beat LLMs working alone, which suggests the gap is in the architecture rather than the training data Do large language models genuinely simulate mental states?. A parallel result outside social reasoning backs this up: splitting reasoning into separate, isolated tool calls unlocked abilities the model already had but that plain prompting couldn't reliably bring out Can modular cognitive tools unlock reasoning without training?. Social simulation research makes the same case from the other side. Agents that only predict plausible behavior are stuck in behaviorism. Modeling a person's belief network and reasoning trace is what makes it possible to ask "what if" questions about how they'd respond Can language models simulate belief change in people?.
The picture is uneven, not empty. In economic games, LLMs show measurable signs of mentalizing, meaning they adjust their strategy based on what they infer the other player believes. But these signs vary sharply by provider and model size, and prompting tricks help some tasks and not others Do LLMs use inferred beliefs to adapt their game strategies?. Data can also stand in for reasoning: models fine-tuned on psychology experiment data predict human decisions better than hand-built cognitive theories do Can language models learn to model human decision making?, though they compress away contextual nuance that humans keep How do language models learn to think like humans?. And matching human outcomes doesn't mean matching human process. LLM groups reproduce a known pattern of human group discussion, but they get there through conformity and early convergence rather than by surfacing what each member uniquely knows Do language model groups mimic human group reasoning patterns?.
The surprising lesson is that "can it reason about minds?" may be the wrong test. Right answers on benchmarks can come from pattern-matching, so the more useful question is whether the system keeps an explicit model of what someone else believes. If you want the philosophical side, there is a defensible middle position that grants LLMs modest, belief-like states without claiming they are conscious Can we defend modest mental attributions to large language models?. That makes "does it track beliefs?" an engineering question rather than a metaphysical one.
Sources 11 notes
Claude 3.7 Sonnet and o1 fail measurably at Decrypto benchmark tasks testing representational change, false belief, and counterfactual reasoning—tasks where they score worse than both humans and simple word-embedding baselines. The evidence suggests formal reasoning optimization actively degrades social reasoning capability.
When semantic content is decoupled from reasoning tasks, LLM performance collapses even with correct rules in context. Models rely on parametric commonsense and token associations rather than formal logical manipulation, constraining reasoning to training distribution semantics.
ChangeMyView and FANTOM benchmarks show LLMs fail at authentic perspective-taking in open-ended scenarios, despite succeeding on structured tasks. Hybrid Bayesian architectures that force explicit belief tracking outperform LLM-alone approaches, suggesting the gap is architectural rather than merely training-based.
LLMs show only slight agreement with human-verified persuasion outcomes (Cohen's κ = 0.079–0.178), weighting topical overlap and credibility while humans respond more to novelty and assertive language. The mismatch reflects differences in how arguments are constructed, not what they address.
Four cognitive tools implemented as sandboxed LLM calls improved GPT-4.1 on AIME2024 from 26.7% to 43.3% without any RL training. Modularity enforces operation isolation that pure prompting cannot guarantee, eliciting pre-existing reasoning capability.
Show all 11 sources
LLM agents remain stuck in behaviorism, producing plausible outputs without internal reasoning structures. Modeling belief networks and reasoning traces enables traceability, counterfactual adaptation, and meaningful policy simulation.
Computational modeling of LLM behavior in economic games revealed clear mentalizing signatures that differed markedly across model providers and sizes, with prompting strategies yielding uneven gains by task. Humans showed both recursive and adaptive mentalization, validating the approach.
LLMs finetuned on psychology experiment data predict human behavior more accurately than theory-driven models in decision tasks, capture individual differences in their embeddings, and transfer learning across tasks without task-specific design.
LLMs trained on psychological data exhibit cognitive phenomena mirroring humans: asymmetric belief updating, event segmentation matching human consensus, and individual-level variation. However, they compress information more aggressively than humans do, sacrificing contextual nuance for statistical efficiency.
LLM groups reproduce the human assembly-bonus asymmetry where discussion helps average members more than top performers, but achieve this through greater conformity, earlier convergence, and less unique information surfacing than human groups.
Both robustness and etiological deflationist arguments beg the question against inflationism. A graded approach ascribing metaphysically undemanding states like beliefs and desires—while withholding consciousness claims—mirrors how we treat non-human animals.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Deflating Deflationism: A Critical Perspective on Debunking Arguments Against LLM Mentality
- Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
- A Systematic Review on the Evaluation of Large Language Models in Theory of Mind Tasks
- Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments
- Language models show human-like content effects on reasoning tasks
- Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
- Determinants of LLM-assisted Decision-Making