AI can ace quiz questions about what a story character believes, yet that score barely predicts whether it reads you better in a real chat.

This explores why AI can pass theory-of-mind (ToM) tests, meaning tests of whether it can track what others believe, want or know, yet doesn't clearly get better at reading real people in conversation. The most direct evidence in the corpus is blunt. Four techniques for improving ToM all raised scores on story-based benchmarks. In multi-turn conversations with humans, the same techniques produced only modest, inconsistent gains, and even where gains appeared they were often too subtle for people to notice Do theory of mind improvements transfer from benchmarks to real conversation?. So a better benchmark score doesn't mean the model understands people better in a way users can feel.

The likely reason is in how these tests are built. A story benchmark hands the model a finished scene: the characters, the hidden information and one correct answer. Real conversation gives the model none of that. It has to work out what this particular person means while the exchange is still unfolding, often from vague or incomplete phrasing. The corpus describes the same pattern in other areas of AI. Automated benchmarks favor tasks that are precisely specified and easy to grade, and this can make AI look both more and less capable than it really is Do automated benchmarks hide what frontier AI systems can really do?. In agent research, systems that win contests still fail at long, real professional workflows, because 'the field optimizes what it measures' Why do agent benchmarks not predict real economic value?. ToM benchmarks may be a case of the same thing: they measure solving a puzzle about minds, not understanding the person you're talking to.

A second missing ingredient is how different people are. One benchmark answer key assumes there is one correct reading of the situation. The MatrAIx project argues that standard benchmarks leave out how different users phrase requests and judge results. It builds simulated user populations to put that variation back into evaluation Can simulated users reveal what offline benchmarks miss?. Social understanding is mostly about this person, not people in general, and static tests can't capture that.

The less obvious point is that training models to be good assistants may itself work against modeling people. One study found that pretrained base models given short samples of real dialogue predicted human behavior more accurately than instruction-tuned assistants prompted with personas. The authors' explanation is that optimizing a model to be an assistant makes it worse at simulating humans Do pretrained models simulate humans better than instruction-tuned assistants?. Another note finds that making a model sound warmer and more empathetic raised error rates by 10–30% Does humanizing AI language reduce model reliability?. Together these suggest that 'seeming socially attuned' and 'actually modeling the person' can pull in different directions.

One caveat: the corpus has only one study that tests ToM transfer to real interaction directly. The broader explanation here comes from evidence about benchmarks versus reality in other areas, so treat it as a strong hypothesis rather than a settled result. If you want to go deeper, start with the ToM transfer study. Then read the open-world evaluation note to see why this kind of gap keeps appearing across AI.