AI models can ace tidy quiz-style tests about what others believe, then fall apart when asked the same thing in plain conversation — why?

This explores why language models can look like they understand other people's beliefs on tidy, structured tests, then stumble when they have to work out what someone thinks in a free-form situation. One caveat first: the collection doesn't isolate multiple-choice as a format. What it does show clearly is a gap between structured, templated tasks and open-ended ones, and the reasons for that gap explain most of what the question asks.

The central finding is that structured theory-of-mind tests can often be passed without tracking anyone's mental state at all. When researchers compared supervised fine-tuning with reinforcement learning on these benchmarks, both reached similar scores. That suggests models are picking up templated story structures and skewed answer distributions, not learning to reason about beliefs Can language models solve ToM benchmarks without real reasoning?. A fixed answer set makes this easier, because the model only has to recognize which option fits the familiar pattern. Open-ended settings such as ChangeMyView persuasion or FANTOM's information-asymmetric conversations remove that support. There, models fall back on surface strategies instead of simulating what a specific person knows or wants. Systems that force explicit belief tracking, using a Bayesian component alongside the LLM, do better, which points to an architectural gap rather than a data gap Do large language models genuinely simulate mental states?.

The same pattern shows up outside social reasoning. Chain-of-thought prompts built from logically invalid examples work almost as well as valid ones. Models learn the shape of reasoning, not the inference itself Does logical validity actually drive chain-of-thought gains?, Why does chain-of-thought reasoning fail in predictable ways?. A benchmark that rewards the right shape will overestimate the underlying skill. The FLEX results make a related point: models often know a fact when asked directly, yet go along with a false assumption built into a question Why do language models accept false assumptions they know are wrong?. Knowing something and using it unprompted in a messy context are different abilities, and structured tests mostly measure the first.

You might expect newer reasoning models to close the gap. On the Decrypto game, which tests false belief and changing mental representations, Claude 3.7 Sonnet and o1 scored worse than older models, worse than humans, and in places worse than simple word-embedding baselines Why do reasoning models fail at theory of mind tasks?. Optimizing for formal reasoning may actually degrade social reasoning.

The most practical consequence is this: four techniques that reliably raised scores on static story benchmarks produced only small, inconsistent gains in real multi-turn conversations with people. Even when gains appeared, users often couldn't notice them Do theory of mind improvements transfer from benchmarks to real conversation?. Surveys of multi-party conversational agents find the benchmarks themselves are mostly synthetic and unvalidated against real group dialogue, and larger models don't reliably do better Can current benchmarks really measure theory of mind in conversational agents?. So the deeper problem isn't just that structured tests hide failures. A whole line of research may be improving models on a target that doesn't track the ability people care about in conversation.