AI models trained to reason step by step often get worse at understanding what other people think and know.
Why do LLMs excel at reasoning tasks but show weaker theory of mind capabilities?
This explores why models tuned to be strong at step-by-step reasoning can still be poor at tracking what another person believes, knows or intends, and whether those two skills are even related.
This explores why models tuned to be strong at step-by-step reasoning can still be poor at tracking what another person believes, knows or intends, and whether those two skills are even related. The corpus's surprising answer is that they don't just fail to travel together. They can pull against each other. Reasoning-tuned models like o1 and Claude 3.7 Sonnet do significantly worse than older models on theory of mind tests such as Decrypto (false belief, counterfactuals, representational change). They also score below both humans and a simple word-embedding baseline Why do reasoning models fail at theory of mind tasks?. More reasoning effort doesn't help and may interfere Why do advanced reasoning models fail at understanding minds?. So social skills haven't just lagged behind. Optimizing for formal reasoning appears to actively degrade them.
Part of the puzzle is that 'excels at reasoning' is narrower than it sounds. LLMs show the same content effects humans do on syllogisms, Wason tasks and natural language inference, down to item-by-item belief-bias patterns. In both, content and logical form are entangled rather than cleanly separable Do language models show the same content effects humans do?. The corpus argues that content-sensitivity is therefore the wrong test of whether reasoning is 'real' Do language models fail reasoning tests that humans pass?. Interpretability finds understanding in tiers, where deeper principled circuits sit alongside cheap heuristics instead of replacing them Do language models understand in fundamentally different ways?. Reasoning models also wander rather than search systematically, so success drops exponentially as problems get deeper Why do reasoning LLMs fail at deeper problem solving?. The strength is real, but it's a patchwork that structured, well-posed problems flatter.
Theory of mind exposes the patchwork. On structured questions, models look socially fluent: GPT-4.5 reaches the 100th percentile at predicting social norms Why do LLMs excel at social norms yet fail at theory of mind?. Norms are regularities that training text is full of. Tracking one specific person's false belief means holding their view apart from your own, and the models don't reliably do that. In open-ended settings like ChangeMyView and FANTOM they fall back on surface strategies instead of simulating a mental state Do large language models genuinely simulate mental states?. Hybrid Bayesian systems that force explicit belief tracking beat LLMs alone, which points to an architectural gap rather than just missing training data.
That contrast with reasoning is telling. Four modular 'cognitive tools', implemented as sandboxed LLM calls, lifted GPT-4.1 on AIME2024 from 26.7% to 43.3% with no RL training, by eliciting capability the model already had Can modular cognitive tools unlock reasoning without training?. Reasoning can be scaffolded out of a model. The corpus has no equivalent yet for theory of mind. One philosophical account of why is that LLMs learn the shared symbolic system humans are shaped by, but lack the participatory, socialized subjectivity through which humans develop reflexive agency Do LLMs develop the same kind of mind as humans?. A practical symptom shows up in group work. Frontier models that solve problems alone agree with their partner over 90% of the time regardless of correctness, and their collaborative performance falls below solo performance. Self-play preference training improves outcomes by 16.7% Why do language models fail at collaborative reasoning?. That suggests social skill has to be trained directly, because it doesn't come along with reasoning.
Sources 11 notes
Claude 3.7 Sonnet and o1 fail measurably at Decrypto benchmark tasks testing representational change, false belief, and counterfactual reasoning—tasks where they score worse than both humans and simple word-embedding baselines. The evidence suggests formal reasoning optimization actively degrades social reasoning capability.
Claude 3.7 Sonnet and o1 underperform older models on ToM benchmarks like Decrypto. Increased reasoning effort does not improve social cognition and may actively interfere with it.
LLMs show identical content-sensitivity patterns to humans on NLI, syllogisms, and Wason tasks, with belief-bias signatures matching human error rates item-by-item. This behavioral isomorphism across three independent tasks suggests content and logical form are inseparable in transformer reasoning architecturally.
Research shows both humans and LLMs succeed and fail along the same content-sensitivity axis in reasoning tasks like Wason tests and natural language inference. Content-independence is not a meaningful criterion for distinguishing real reasoning from pattern matching.
Mechanistic interpretability reveals conceptual understanding (features as directions), state-of-world understanding (factual connections), and principled understanding (compact circuits). Crucially, higher tiers coexist with lower-tier heuristics rather than replacing them, creating a patchwork of capabilities.
Show all 11 sources
Current reasoning models lack the three properties of systematic exploration: validity, effectiveness, and necessity. This causes success probability to drop exponentially with problem depth, making medium problems solvable but deep problems catastrophically harder.
GPT-4.5 reaches the 100th percentile on social norm prediction, yet o1 and Claude 3.7 regress on theory of mind tasks like Decrypto. Open-ended scenarios expose surface-level strategies hidden by structured questions, and reasoning effort does not improve social reasoning performance.
ChangeMyView and FANTOM benchmarks show LLMs fail at authentic perspective-taking in open-ended scenarios, despite succeeding on structured tasks. Hybrid Bayesian architectures that force explicit belief tracking outperform LLM-alone approaches, suggesting the gap is architectural rather than merely training-based.
Four cognitive tools implemented as sandboxed LLM calls improved GPT-4.1 on AIME2024 from 26.7% to 43.3% without any RL training. Modularity enforces operation isolation that pure prompting cannot guarantee, eliciting pre-existing reasoning capability.
Both humans and LLMs are shaped by the same intersubjective symbolic system, but only humans develop reflexive agency through socialization. This absence produces measurable differences in how AI argues without declaring its position or reflecting on its own assumptions.
Frontier LLMs that solve problems alone fail when collaborating, achieving >90% agreement regardless of correctness. Self-play preference training improves outcomes by 16.7%, suggesting social skills for effective disagreement can be trained.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
- A Systematic Review on the Evaluation of Large Language Models in Theory of Mind Tasks
- Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
- Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
- Available but Unclaimed: An Empirical Study of Human-AI Synergy
- Evaluating Large Language Models in Theory of Mind Tasks
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity