Can AI ace tests of 'reading minds' just by spotting patterns, without actually tracking what others believe?
Can language models solve theory of mind benchmarks without genuine mental state reasoning?
This explores whether AI models can score well on tests built to measure whether they understand what other people believe, want or know ("theory of mind") without actually reasoning about those mental states, and what the collection suggests is happening instead.
This explores whether language models can pass theory-of-mind tests (tasks like 'where will Sally look for the marble she didn't see moved?') without doing anything resembling real reasoning about other minds. The collection's short answer is yes, and the evidence for it is unusually direct. When researchers trained models on these benchmarks, plain supervised fine-tuning, which just imitates correct answers, matched reinforcement learning, which is supposed to build reasoning. That suggests the tests reward spotting templates and skewed answer patterns more than tracking beliefs Can language models solve ToM benchmarks without real reasoning?. Many of these benchmarks are also built from synthetic, scripted stories that look nothing like real group conversation, and making the model bigger doesn't reliably raise scores, so no one can yet say whether a high score means anything outside the test Can current benchmarks really measure theory of mind in conversational agents?.
The clearest sign of surface strategy is where the scores break. Models do well on structured, textbook-style false-belief puzzles, then stumble once the setting gets open-ended, such as following who knows what in a messy multi-party chat or reading a persuasion thread on ChangeMyView Do large language models genuinely simulate mental states?. That pattern matches a wider finding about chain-of-thought: models reproduce the shape of reasoning they've seen in training, and performance drops predictably when the problem moves away from familiar forms Does chain-of-thought reasoning reveal genuine inference or pattern matching?. Related work goes further and finds that reasoning traces with invalid logical steps help almost as much as valid ones. The written 'thinking' works more as a performance than as a record of how the answer was reached Do reasoning traces show how models actually think?.
The less obvious finding is that more reasoning can make social understanding worse. Models tuned for step-by-step reasoning, like o1 and Claude 3.7 Sonnet, scored below older models on the Decrypto game's false-belief and perspective tasks. On some of them they fell below simple word-embedding baselines Why do reasoning models fail at theory of mind tasks?. They write longer traces that don't help, and what they learn doesn't carry over to near-identical scenarios. A method called ThoughtTracing did better with shorter traces by keeping several competing guesses about what each person believes and updating them as evidence arrives Why do reasoning models struggle with theory of mind tasks?. The suggestion is that social reasoning isn't a longer version of math reasoning. It means juggling several models of other minds at once, not deriving one answer in a chain. Hybrid systems that force explicit belief tracking beat LLMs working alone, which points to an architecture gap rather than a lack of training data Do large language models genuinely simulate mental states?.
Two caveats keep this from becoming 'LLMs have no grasp of minds at all.' First, models fine-tuned on psychology experiment data predict real human decisions better than purpose-built cognitive theories, so they capture something about how people behave, even if not through explicit belief reasoning Can language models learn to model human decision making?. Second, some philosophers argue that 'genuine' is doing heavy lifting here. A modest view, which credits LLMs with simple beliefs and desires but not consciousness (roughly how we treat animals), holds up against the standard debunking arguments Can we defend modest mental attributions to large language models?. The models' grip on their own minds looks similarly shallow: their self-reports are unstable and shift under conversational pressure How well do language models understand their own knowledge?.
So the takeaway isn't only that models cheat on theory-of-mind tests. Today's tests can't tell pattern matching from mind-reading, and the one tool we expected to close that gap, extended reasoning, seems to work against it. If you want to find out whether a model understands other people, a higher benchmark score won't tell you. Check whether it can keep track of several people's beliefs in a messy, unfamiliar conversation.
Sources 10 notes
Supervised fine-tuning matches reinforcement learning performance on ToM tasks, suggesting models exploit structural vulnerabilities rather than develop genuine reasoning. Distribution biases and templated artifacts allow surface-level pattern recognition to achieve competitive generalization.
A survey of multi-party conversational agents finds that current theory of mind benchmarks use synthetic datasets that don't reflect realistic group conversations, and scaling model size doesn't reliably improve performance. Real-world validity remains unsettled.
ChangeMyView and FANTOM benchmarks show LLMs fail at authentic perspective-taking in open-ended scenarios, despite succeeding on structured tasks. Hybrid Bayesian architectures that force explicit belief tracking outperform LLM-alone approaches, suggesting the gap is architectural rather than merely training-based.
CoT works by constraining models to reproduce familiar reasoning patterns from training, not by enabling novel symbolic reasoning. Performance degrades predictably under distribution shifts—the signature of imitation rather than capability emergence.
LLM reasoning traces perform as persuasive appearances rather than reliable explanations of computation. Invalid logical steps perform nearly as well as valid ones, and corrupted traces generalize comparably, showing that semantic correctness is not what produces the performance gains.
Show all 10 sources
Claude 3.7 Sonnet and o1 fail measurably at Decrypto benchmark tasks testing representational change, false belief, and counterfactual reasoning—tasks where they score worse than both humans and simple word-embedding baselines. The evidence suggests formal reasoning optimization actively degrades social reasoning capability.
Reasoning models fail to outperform vanilla LLMs on theory of mind tasks, produce longer but unhelpful traces, and show no generalization to similar scenarios. ThoughtTracing's success using shorter Bayesian hypothesis tracking suggests social reasoning demands simultaneous multiple-model maintenance, not sequential derivation.
LLMs finetuned on psychology experiment data predict human behavior more accurately than theory-driven models in decision tasks, capture individual differences in their embeddings, and transfer learning across tasks without task-specific design.
Both robustness and etiological deflationist arguments beg the question against inflationism. A graded approach ascribing metaphysically undemanding states like beliefs and desires—while withholding consciousness claims—mirrors how we treat non-human animals.
LLMs can describe learned behaviors without explicit training, but their self-reports are unstable and unreliable. Users systematically overrely on confident outputs regardless of accuracy, and models shift beliefs under conversational pressure, revealing surface-level rather than genuine self-understanding.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- A Systematic Review on the Evaluation of Large Language Models in Theory of Mind Tasks
- Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
- Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- Deflating Deflationism: A Critical Perspective on Debunking Arguments Against LLM Mentality
- LLM Reasoning Is Latent, Not the Chain of Thought
- Evaluating Large Language Models in Theory of Mind Tasks