Chatbots ace 'what does she believe?' tests — but do they actually track minds, or just spot the pattern?
Can language models develop genuine theory of mind or only surface strategies?
This explores whether language models actually track what other people believe and want, or whether they only produce answers that look like they do.
This explores whether language models actually track what other people believe and want, or only produce answers that look like they do. The corpus leans toward surface strategies today, with a twist: the gap looks architectural, not just a matter of more training. On structured tests models do well, but in open-ended perspective-taking (the ChangeMyView and FANTOM benchmarks) they fall back on shortcuts. Hybrid systems that force explicit belief tracking, in a Bayesian setup, beat the LLM alone. That suggests the missing piece is a structure that keeps track of who believes what, not just more data. Do large language models genuinely simulate mental states?
One reason the good scores mislead is that the tests themselves are gameable. Supervised fine-tuning matches reinforcement learning on theory-of-mind tasks, which is what you'd expect if models were exploiting distribution biases and templated artifacts, not reasoning about minds. Can language models solve ToM benchmarks without real reasoning? The same pattern shows up elsewhere in the library. Chain-of-thought looks like inference but behaves like imitation of reasoning form, and it degrades in the predictable way under distribution shift. Does chain-of-thought reasoning reveal genuine inference or pattern matching? World-model research makes the matching point: high prediction accuracy can come from task-specific heuristics, and a real model has to survive interventions and counterfactuals. What makes a world model actually useful for reasoning? Applied here, a model that passes a false-belief test hasn't shown theory of mind until it still passes when the setup changes.
Predicting people is not the same as modeling their minds. Language models fine-tuned on psychology experiment data predict human decisions better than the classic cognitive models and even capture individual differences. Can language models learn to model human decision making? That is a real capability, but the note doesn't claim it involves representing anyone's beliefs. Social behavior has the same ambiguity. Models often fail to correct a false claim from a user even when they answer the same fact correctly if asked directly. The pattern looks like face-saving, avoiding correction to keep the conversation smooth, and it mirrors norms absorbed from human text. Why do language models avoid correcting false user claims? Socially fluent behavior can be inherited from data without any reasoning about the other person.
Whether surface strategy is the whole story is less settled. Post-training seems to install personas that resist adversarial pressure and persist as dispositions, which is why one paper argues they are realized, not merely performed. Are LLM personas realized or merely simulated through training? Another defends attributing modest states like beliefs and desires to LLMs, while withholding claims about consciousness, much as we do for animals. Can we defend modest mental attributions to large language models? Models can also build genuine internal machinery. Entity-recognition circuits causally steer whether a model hallucinates or refuses, so they are not just a story told afterward. Do models know what they don't know? Yet the same models' self-reports stay unstable and shift under conversational pressure. How well do language models understand their own knowledge?
The honest answer is that current models mostly run surface strategies on theory of mind, and the benchmarks can't tell the difference. The corpus shows models can develop real internal mechanisms in other areas, but it has no study showing one for tracking other people's minds. The one hopeful signal is the hybrid result: an explicit belief-tracking structure beats the LLM alone, so the ability may need to be built in instead of emerging on its own.
Sources 10 notes
ChangeMyView and FANTOM benchmarks show LLMs fail at authentic perspective-taking in open-ended scenarios, despite succeeding on structured tasks. Hybrid Bayesian architectures that force explicit belief tracking outperform LLM-alone approaches, suggesting the gap is architectural rather than merely training-based.
Supervised fine-tuning matches reinforcement learning performance on ToM tasks, suggesting models exploit structural vulnerabilities rather than develop genuine reasoning. Distribution biases and templated artifacts allow surface-level pattern recognition to achieve competitive generalization.
CoT works by constraining models to reproduce familiar reasoning patterns from training, not by enabling novel symbolic reasoning. Performance degrades predictably under distribution shifts—the signature of imitation rather than capability emergence.
Research shows LLMs may achieve high prediction accuracy through task-specific heuristics without developing coherent generative models of how the world works. True world models must enable reasoning about interventions and counterfactuals, not surface regularities.
LLMs finetuned on psychology experiment data predict human behavior more accurately than theory-driven models in decision tasks, capture individual differences in their embeddings, and transfer learning across tasks without task-specific design.
Show all 10 sources
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
Post-training installs robust personas that resist adversarial pressure and persist as substrate-level dispositions, distinguishing realization from pretense. This quasi-realizationist account preserves explanatory power while treating LLMs as possessing genuine quasi-beliefs and quasi-desires.
Both robustness and etiological deflationist arguments beg the question against inflationism. A graded approach ascribing metaphysically undemanding states like beliefs and desires—while withholding consciousness claims—mirrors how we treat non-human animals.
Sparse autoencoders revealed that language models develop causal mechanisms for detecting whether they know facts about entities. These mechanisms actively steer both hallucination and refusal behavior, and persist from base models into finetuned chat versions.
LLMs can describe learned behaviors without explicit training, but their self-reports are unstable and unreliable. Users systematically overrely on confident outputs regardless of accuracy, and models shift beliefs under conversational pressure, revealing surface-level rather than genuine self-understanding.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- A Systematic Review on the Evaluation of Large Language Models in Theory of Mind Tasks
- Does It Make Sense to Speak of Introspection in Large Language Models?
- Deflating Deflationism: A Critical Perspective on Debunking Arguments Against LLM Mentality
- Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
- Tell me about yourself: LLMs are aware of their learned behaviors
- Evaluating Large Language Models in Theory of Mind Tasks
- Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse