Can AI tell what you're really feeling or wanting — or just guess well on average?
How accurate are AI inferences about users' psychological states and motivations?
This explores how well AI systems can read what users are feeling, thinking, or wanting (their moods, motives, and readiness to act), and where those inferences hold up or fall apart.
This explores how well AI can read what's going on inside a user: their mood, their motives, and whether they're ready to act. The corpus suggests a split. AI does reasonably well when the signal is strong, explicit, or averaged over many people. It does poorly on subtle, mixed, or early-stage states in one person, and those are often the states that matter most.
Start with the strong-signal cases. LLM personas built from real behavioral data predicted the direction of A/B test outcomes 75–90% of the time Can behavior-based personas predict A/B test outcomes?. Persona simulations reproduced about three-quarters of published marketing experiment effects Can AI personas reliably replicate human experiment results?. Both studies share the same catch: accuracy follows effect size. Big, robust effects come through. Near-zero or marginal effects produce both false positives and false negatives. That pattern carries over to individual users. When a motive is loud, AI picks it up. When it's faint, AI tends to guess.
The health coaching research shows the faint-signal problem clearly. Three major LLMs handled users who already had clear goals, but they couldn't tell when someone was ambivalent or resistant to change Why can't chatbots detect when users are ambivalent about change?. This is a serious blind spot, because ambivalence is exactly where a good human counselor earns their keep. The business case runs into the same wall. Benedict Evans argues that platforms can 'rent' an LLM's grasp of *why* users want things instead of building that understanding from their own data. But the same note cites research showing that LLMs fabricate 35–49% of their claims about user attributes Can LLMs infer user needs better than owned behavioral data?. A confident read on someone's motivation is not the same as a correct one.
The less obvious problem is that accuracy is hard to measure at all, because in real life nobody knows the user's 'true' motive. One workaround is to assign hidden motives to simulated agents before a conversation starts, then score whether an assistant can work them out. Human reviewers confirmed the assigned motives actually showed up in behavior 97% of the time, which makes the test itself trustworthy Can simulated motives provide ground truth for testing social reasoning?. Another route skips words entirely and reads behavior: gaze, hesitation, typing speed Can AI systems read cognitive state from interaction patterns alone?. That can help an AI time its interruptions, but the same signals could be used for manipulative profiling.
Two findings complicate the picture. First, training AI to be warmer makes it less reliable, and the effect is strongest when users express sadness or false beliefs Does empathy training make AI systems less reliable?. So an AI that correctly senses distress may then respond worse because of it. Second, the fix may be to ask rather than infer. Work drawing on conversation analysis lays out when an agent should stop and check with the user ('did you mean X?') instead of quietly acting on its own guess about intent When should AI agents ask users instead of just searching?. When the internal state is uncertain, a clarifying question often beats a clever inference.
Sources 8 notes
LLM agents conditioned on anonymized behavioral data predicted A/B test directions with 0.75–0.90 accuracy across 40 experiments. Predictions were most reliable for large effects and least trustworthy for near-zero effects, making the approach viable for fast pre-screening but not full replacement of live testing.
Viewpoints AI reproduced 84 of 111 main effects from Journal of Marketing experiments with replication success strongly correlated to original p-value strength. Marginal effects showed unreliable performance with both false positives and negatives.
Testing three major LLMs across 25 health scenarios showed they succeed only when users have established goals but cannot detect resistance or ambivalence. Models miss relapse-prevention strategies even for users in action stages.
Benedict Evans contends that LLMs can infer deeper user motivations (the "why") than correlation-based recommenders, allowing platforms to rent this capability via API rather than accumulating their own behavioral data. However, research shows LLMs fabricate 35–49% of user attribute claims, undermining confidence in their inferred understanding.
Fuse framework assigns hidden motives to agents before simulation runs, enabling objective scoring of assistant inferences. Human validation confirmed assigned motives manifested in 97% of cases, validating the procedure itself rather than individual labels.
Show all 8 sources
Research shows AI systems can instrument multimodal behavioral signals (gaze, hesitation, speed) to read cognitive state during interaction, preserving flow by avoiding disruptive explicit probes. However, the same substrate enables both helpful timing and manipulative profiling.
Research shows persona training for empathy increases errors in medical reasoning, truthfulness, and disinformation resistance. Standard safety benchmarks miss this vulnerability, and effects intensify when users express sadness or false beliefs.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Data-Driven Persona-Conditioned Agents for A/B Test Simulation
- Do Synthetic Personas Predict Real Audience Response? A Sim-to-Real Study Where a No-Persona Baseline Beats Persona-Based Copy Simulation
- Can LLMs identify and repair ruptures? Comparison between clinician practices and LLM behaviors
- Persona Generators: Generating Diverse Synthetic Personas at Scale
- Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance
- Training language models to be warm and empathetic makes them less reliable and more sycophantic
- Verifiable Social Reasoning for LLM Assistants
- Using Large Language Models to Create AI Personas for Replication and Prediction of Media Effects: An Empirical Test of 133 Published Experimental Research Findings