Do theory of mind improvements transfer from benchmarks to real conversation?

The paper's authors test whether theory of mind (ToM) enhancement techniques that raise scores on static, story-based ToM benchmarks actually help large language models in real human-AI interaction, and find that they often do not. Four representative techniques, two prompting methods (Foresee and Reflect, and Perspective Taking) and two fine-tuning methods (supervised fine-tuning and reinforcement learning), are applied to GPT-4o and Llama-3.1-8B; the authors confirm all of them raise scores on existing third-person ToM benchmarks. But when the same model variants are tested in multi-turn, first-person interaction across nine tasks, the authors report "a significant gap between how models perform on static, story-based ToM benchmarks and their actual capabilities in dynamic, interactive scenarios," concluding plainly that "improvements on static benchmarks do not always translate to better performance in dynamic HAI interactions."

The mechanism is a paradigm shift the authors build deliberately. Standard ToM benchmarks position the model as a third-person observer of a fixed narrative, scored only by accuracy against a correct option. The authors instead place the model as a first-person participant in a multi-turn dialogue with a human, scored by a broader metric schema that can include goal completion rate and human satisfaction. Nine tasks across four benchmarks are split, drawing on Bales's (1950) distinction between task and socio-emotional group processes, into goal-oriented scenarios (math, code, document editing) and experience-oriented scenarios (counseling, emotional support), plus a crowdsourced user study to check that benchmark results track genuine human perception. The authors add a finding beyond the headline gap: where ToM methods did help, in experience-oriented tasks, the improvement was often "too subtle to cross a user's perceptual threshold" — measurable in benchmark terms but not reliably noticed by the people interacting with the model.

This sharpens a line of doubt already running through nearby notes about what ToM benchmarks measure. Do large language models genuinely simulate mental states? showed that open-ended questions expose ToM failures that multiple-choice formats hide; this paper extends that logic from base capability to enhancement techniques, showing the same format gap applies to methods specifically built to improve ToM. It also bears on Can language models solve ToM benchmarks without real reasoning?, since the SFT and RL techniques questioned there (Sclar et al.'s ExploreToM fine-tuning and Lu et al.'s ToM-RL) are the exact methods tested here, now shown to raise benchmark numbers without reliably raising interactive performance. And it adds an interaction-first case to Can current benchmarks really measure theory of mind in conversational agents?, which argued from the survey side that ToM benchmarks are synthetic and insufficient for group-chat agents; this paper reaches a parallel conclusion empirically, for one-on-one HAI dialogue rather than multi-party settings.

The excerpt does not establish why the gap occurs beyond naming it a "socio-technical gap" between benchmark performance and real-world competence, and it tests only two base models (GPT-4o, Llama-3.1-8B), so it says nothing about whether newer or larger models close the gap. The human-study preference for ToM-enhanced models is also described as "consistent but not statistically significant." The implication the authors draw, and the evidence supports at this strength, is that static ToM benchmark gains should not be read as evidence of interaction readiness, and that developing socially aware LLMs requires interaction-based evaluation alongside, not instead of, story-based tests.