Does Theory of Mind Improvement Really Benefit Human-AI Interactions? Empirical Findings from Interactive Evaluations

Improving the Theory of Mind (ToM) capability of Large Language Models (LLMs) is crucial for effective social interactions between these AI models and humans. However, the existing benchmarks often measure ToM capability improvement through story-reading, multiple-choice questions from a third-person perspective, while ignoring the first-person, dynamic, and open-ended nature of human-AI (HAI) interactions. To directly examine how ToM improvement techniques benefit HAI interactions, we first proposed the new paradigm of interactive ToM evaluation with both perspective and metric shifts. Next, following the paradigm, we conducted a systematic study of four representative ToM enhancement techniques using both four benchmarks based on real-world scenarios and a user study, covering both goal-oriented tasks (e.g., coding, math) and experience-oriented tasks (e.g., counseling). Our findings reveal that improvements on static benchmarks do not always translate to better performance in dynamic HAI interactions. This paper offers critical insights into ToM evaluation, showing the necessity of interaction-based assessments in developing next-generation, socially aware LLMs for HAI symbiosis.

Introduction. Theory of Mind (ToM) denotes the cognitive capacity to attribute unobservable mental states (e.g., beliefs, intentions, emotions), which is essential for social interaction (Chen et al., 2025a; Sarıta ̧s et al., 2025; Strachan et al., 2024). As a foundational component of social cognition, ToM has been recognized as a core social intelligence skill that advanced LLMs should obsess to improve their interactions with humans and ultimately achieve the target of human-AI symbiosis (Street, 2024).

To nurture LLMs’ ToM capability, the cornerstone is to understand its capability levels and benefits led by improvement methods with appropriate and sufficient evaluation. To achieve the goal, the existing dominant methods are static, task-based assessments in a story-question-option format, an approach derived from classic false-belief tests like the Sally-Anne task, such as the one by Kosinski (Kosinski, 2024). Following the design, subsequent benchmarks such as HiToM (Wu et al., 2023) and ToMBench (Chen et al., 2024) have increased the complexity and diversity of these tests. More recent benchmarks, such as Explore- ToM (Sclar et al., 2025), leverage adversarial methods to further increase the diversity of problems and reduce the risk of memorizing benchmarks in the LLM training process. However, these third-person, story-reading benchmarks with accuracy as the only standard fail to ground ToM evaluation in the real-world context of Human- AI (HAI) interaction and collaboration. In HAI scenarios, LLMs are supposed to leverage their ToM capability to perform first-person perspective actions in response to dynamic and sometimes open-ended user requests with diverse targeted metrics. The mismatch of task natures creates a critical socio-technical gap (Liao and Xiao, 2023) between benchmark performance and real-world competence, where the ToM benchmark result improvements may not lead to sensible benefits for human-LLM interactions. To reveal the gap and guide future ToM evaluation, our research examines how these benchmark improvements are transformed into real-world values through a new paradigm of interactive ToM evaluation. We first shift the ToM task from a static, third-person perspective into a dynamic, open-ended, and first-person one, where the LLM agent engages in multi-turn conversations across diverse and real-world scenarios. Next, drawing from cognitive science (Epstein, 1998; Amir et al., 2025; Bales, 1950), we classify these scenarios into two primary categories based on the evaluation objective: goal-oriented tasks (e.g., math, code) and experience-oriented tasks (e.g., counseling, healthcare). Then, we simulate real-world HAI interactions in nine tasks under the two types of scenarios and leverage task-specific metrics (e.g., accuracy, success rate) to evaluate the performance of LLMs with ToM enhancement techniques, including both prompt-based techniques and finetuningbased ones. By aggregating the results from four benchmarks, we comprehensively assess the effectiveness of ToM enhancement techniques across nine domains that align well with actual user requirements. Furthermore, we also conduct a crowdsourcing user study to support our findings, ensuring the results reflect genuine human perceptions. Our rigorous evaluation reveals three key insights regarding current ToM enhancement techniques: (i) A Performance Gap in Evaluation: There is a significant gap between how models perform on static, story-based ToM benchmarks and their actual capabilities in dynamic, interactive scenarios, showing that current evaluation methods are insufficient for measuring readiness for HAI collaboration. (ii) A Failure to Generalize: ToM enhancement techniques improve a model’s performance in experience-oriented tasks but fail to generalize this success to goal-oriented tasks, separating the capability requirements in various real-world scenarios. (iii) A Gap in User Perception: The modest gains from current ToM methods are often too subtle to cross a user’s perceptual threshold, meaning the improvements measured in benchmarks do not translate into a meaningfully better user experience. Our contributions include:

• We shift ToM evaluation from static tests to dynamic, real-world HAI interactions. • We assess ToM enhancement methods in goaland experience-oriented scenarios via simulated interactive benchmarks and user studies.

• We identify critical limitations in current ToM enhancement methods and provide insights for future research.

Related work. Assessment of ToM ToM assessment in LLMs has primarily relied on story-based benchmarks extending classical psychological tests (Sarıta ̧s et al., 2025; Nguyen, 2025). Early benchmarks like ToMi and Hi-ToM expanded this approach with diverse narratives and higher-order reasoning (Le et al., 2019; Wu et al., 2023). Subsequent efforts improved protocols, such as ToMChallenges’ varied templates and FANTOM’s detection of “illusory ToM” in dialogues (Ma et al., 2023; Kim et al., 2023). Concurrently, BigToM and Open- ToM broadened the scope to include mental states like percepts and emotions (Gandhi et al., 2023; Xu et al., 2024). Recent works address domain-specific reasoning (NegotiationToM) and systematic coverage (ToMBench) (Chan et al., 2024; Chen et al., 2024), alongside novel data generation techniques utilizing search algorithms or information asymmetry (Sclar et al., 2025; Shinoda et al., 2025). However, most benchmarks remain passive evaluations, positioning models as observers. This limitation results in only a partial view of ToM competence, motivating our interactive protocol development.

Enhancement of ToM Recent research studies enhancing LLM ToM capabilities through three primary categories (Chen et al., 2025a). 1) Prompt engineering guides reasoning without retraining (Wang and Zhao, 2024; Hou et al., 2024). For instance, FaR prompts reflection on predicted story evolutions (Zhou et al., 2023a), while SimToM filters context to strictly match a character’s perception (Wilf et al., 2024). 2) Fine-tuning adapts models using specialized datasets. Approaches include ToM-RL, which utilizes reinforcement learning (e.g., GRPO) (Lu et al., 2025), and Explore- ToM, which applies supervised fine-tuning on diverse, challenging benchmarks (Sclar et al., 2025). 3) External module integration augments models via specialized components (Huang et al., 2024; Chen et al., 2025b; Sarangi et al., 2025; Kim et al., 2025). For example, AutoToM refines agent models via inverse planning (Zhang et al., 2025).

Method. ToM evaluation in existing benchmarks is typically operationalized through a static, story-questionoption format. Formally, given a story S = {s1, s2, . . . , sn} and a question Q, the model must select the correct answer from a candidate set O = {o1, . . . , ok}, where only one option ocorrect is correct:

2.2 Our Paradigm: Pivot ToM Evaluation to Interactive HAI Settings than merely select an option in a fixed text. Accordingly, we study ToM in human–AI interaction (HAI), where an LLM agent A interacts with a human H through a multi-turn dialogue. Let D1:t = (u1, . . . , ut) ∈D denote the dialogue history up to turn t, where each utterance ui is produced by either H or A. Given a task T ∈G, the agent first infers a latent mental state Perspective In static benchmarks, the model acts as a third-person observer, reasoning about a fixed narrative world. In interactive HAI settings, the model becomes an active participant, required to anticipate, adapt to, and influence the human’s mental state throughout interactions from the firstperson perspective.

Metrics While static settings evaluate models solely by accuracy over predefined answers, interactive HAI settings require a richer metric. In our formulation, evaluation follows the general schema MΓ, which can incorporate metrics such as goal completion rate and human satisfaction. Ultimately, this paradigm shift reframes the evaluation of ToM from a measure of static reasoning accuracy to a measure of dynamic collaborative effectiveness.

Existing methods for enhancing the ToM capabilities of LLMs can be broadly categorized into three approaches: prompt engineering, fine-tuning, and external module integration. As our primary goal is to study how well the existing techniques can improve model ToM capability rather than to build new AI systems with multiple modules, we select methods from the first two categories. Specifically, Foresee and Reflect (FaR) (Zhou et al., 2023a), Perspective Taking (PT) (Wilf et al., 2024), Supervised Fine-tuning (SFT) (Sclar et al., 2025), and Reinforcement Learning (RL) (Lu et al., 2025) to conduct our experiments. A systematic review and our selection criteria are in Appendix A.1. A key challenge is that while our HAI interaction setting requires first-person dialogues, most existing ToM methods are designed for third-person, multiple-choice tasks. We therefore adapt the selected methods to be suitable for direct interaction, as illustrated in Figure 2. For prompting methods, we retain their core principles (e.g., reflection and perspective-taking) and reformulate the prompts for a first-person conversational context. For finetuning methods, we convert the training data to a first-person perspective by replacing the protagonist’s name with “I”. We then apply these adapted methods to two widely used base models, GPT-4o and Llama-3.1-8B, to create our suite of test models. Note that the GPT-RL model is not included due to fine-tuning limitations. Appendix A.3 shows that these techniques improve both base models’ performance on existing ToM benchmarks. Next, model variants with these techniques are applied in our interactive evaluations to verify whether improvements on existing benchmarks translate to tangible benefits in dynamic HAI interaction.

Before experiments, we identify HAI interaction scenario types to guide what datasets and metrics should be used for a comprehensive evaluation. Interaction Process Analysis (IPA) shows that human group interaction reliably bifurcates into task and socio-emotional processes (Bales, 1950). Driven by this classic theory, we classify the HAI scenarios into two distinct categories: goal-oriented and experience-oriented.

Goal-oriented tasks Tasks in this categoty involve users leveraging an LLM as an assistant to accomplish a specific and measurable objective (e.g., code generation and document editing). Prior research suggests that ToM can improve task accuracy by strengthening the coordination protocol between the user and the model (Engel et al., 2014).

Discussion. Takeaway 1: ToM enhancement techniques fail to consistently improve goal-oriented task performance across diverse domains. base models may introduce statistically meaningful safety and ethical trade-offs, whereas stronger models such as GPT-4o appear more robust.

Takeaway 2: ToM enhancements boost empathy and user experience, although fail to support goal achievement. Furthermore, SFT and RL can amplify safety and ethical risks.

Our human evaluation reveals a consistent but not statistically significant preference for models with ToM enhancement techniques, aligning with the results of experience-oriented benchmarks. Across two model families, we can see that models based on prompt-based methods (FaR and PT) To summarize, we consider that the enhanced models still lack of sufficient ToM capability to capture users’ nuanced intention from interactions. For example, we noticed that all model variants, including the best model ranked by the user (i.e., Llama- 3.1-8B-FaR), fail in suggesting more diverse methods to facilitate their sleep problem (Figure 5-right). Beyond direct instructions for models, the underlying adaption to user preferences and scenarios, such as conversation styles, also poses higher requirements on the ToM capability of LLMs. Furthermore, our results show that no one model variant can achieve the best across scenarios, implying their limited ToM capability for dynamic and diverse HAI interactions (Table 11 in Appendix B.2). Taken together, these results suggest that current ToM-enhancement methods do not yet yield substantial improvements in perceived user experience in realistic human-AI interactions, highlighting the need for more realistic evaluation frameworks and for new approaches to ToM improvement.

Takeaway 3: While the ToM methods benefit HAI, realizing their full potential requires enhancing dynamic user understanding.

HAI symbiosis poses new challenges for ToM. Our evaluation framework marks a methodological shift designed to assess ToM for the challenges of HAI symbiosis. This new perspective reveals a significant performance gap. We observe that the methods that improve story-reading benchmark performances only show limited and inconsistent benefit in our interactive evaluation. This gap highlights the necessity and importance of our framework for gaining a complete picture of a model’s true capabilities. It shows that excelling at test-taking tasks does not guarantee readiness for interactive collaboration. Therefore, it is essential to complement existing benchmarks with dynamic and interactive evaluations in HAI contexts.

Enhanced ToM fails to generalize from assistance to companionship. Our findings show that ToM-enhancement methods improve performance in experience-oriented scenarios but fail in goaloriented tasks. This performance difference appears to stem from the distinct nature of these two task categories. Experience-oriented tasks are largely defined by their focus on interpersonal dynamics and responding to affective states like emotions and desires. In contrast, goal-oriented tasks can require understanding users’ intention progress and underlying knowledge states for task accomplishment. This suggests that ToM proficiency in one type of task may not guarantee success in the other, as each emphasizes different aspects of user understanding. This capability gap highlights that future research needs diverse and real-world benchmarks that assess a full spectrum of abilities from empathetic support to goal-driven collaboration.

Users require threshold-crossing ToM improvements. Comprehensive ToM capability is fundamental to achieving true HAI symbiosis, as it is the capability that transforms models from passive text processors into proactive, collaborative partners.

Conclusion. In this paper, we re-examine the effectiveness of ToM enhancement by moving beyond static, thirdperson evaluation to dynamic, open-ended, firstperson human-AI interaction. Built on the HAI symbiosis paradigm, we organize application scenarios into goal-oriented and experience-oriented tasks, and conduct both simulated benchmarking and a user study to assess current ToM enhancement methods in these settings. By systematically comparing representative enhancement methods with two distinct base models on nine tasks, our results show that existing evaluation protocols and method designs remain misaligned with real-world human needs and fail to deliver meaningful improvements in user experience. We hope this work provides a guidance for future human-centric ToM evaluation and the development of socially intelligent AI systems for human-AI symbiosis.

Limitations. Our research is not without limitations. First, we only tested a limited coverage of ToM enhancement methods. The reasons hindering us from including other methods include (1) it is challenging to ask users to compare many model responses generated by variants with different techniques at the same time and (2) many ToM-enhancement methods are hard to be adapted to HAI scenarios. For example, TimeToM (Hou et al., 2024) designed an algorithm to model characters’ movements and mental states from the third-person perspective, which is challenging to be generalized to open-ended HAI interactions from a first-person perspective. As a result, we finally kept the four carefully selected representative techniques that can be adapted to HAI scenarios from both prompt engineering-based and finetuning-based methods. Second, our research can be extended in more potential HAI scenarios. Currently, we only gathered data from nine scenarios in four datasets, including math problem solving, collaborative writing, mental counseling, and emotional support. It is limited by the availability of data from other highly related scenarios, such as customer support. We sincerely hope to further extend our research when more real-world data is available.