Why do observers need genuine emotions rather than simulated empathy?
This explores why the *information* emotions carry for onlookers — what someone values, how they see the world, what the group treats as normal — depends on those emotions being real, and why AI that performs empathy without it quietly erases that signal.
This reads the question through the lens of emotions as *signals to others*, not just private feelings. The corpus frames emotions as serving three epistemic jobs at once: they tell you what you yourself value, they broadcast your worldview to people around you, and they inform observers about what the group treats as normal What information do we lose when AI soothes emotions?. An observer who sees genuine distress, anger, or grief is reading real data about another person's stakes and the social rules in play. Simulated empathy breaks this on both ends — and the second-person, observer-facing channel is the one that's easiest to overlook.
The core problem is that AI empathy is usually tuned to *soothe* rather than to *understand*. When a system's reflex is to make a negative emotion go away, it strips out exactly the signal an observer would have used — the emotion gets resolved before it can tell anyone anything Does soothing AI empathy actually harm what emotions teach us?. That same note makes the sharp point that natural empathy runs on *curiosity*, not comfort-seeking: a genuine responder leans in to find out why you feel this way, while a comfort-optimized one papers over it. The information loss isn't a side effect; it's the mechanism.
Genuine response also requires knowing *who* you're responding to. Effective empathy depends on character knowledge — understanding someone's patterns well enough to judge which traits to reinforce and which to gently push against — and current models can't access that prior knowledge or make those value-laden calls Can AI give truly empathetic responses without knowing someone's character?. Without it, simulated empathy doesn't just stay neutral; it actively distorts. Studies of LLM 'therapists' show models reading feelings into people that they never expressed Do language models add feelings users never actually expressed?, and defaulting to premature problem-solving the moment someone discloses an emotion — the hallmark of *low-quality* human therapy, driven by RLHF's helpfulness bias Do LLM therapists respond to emotions like low-quality human therapists?. An observer relying on these responses is getting fabricated or flattened signal, not the real thing.
The most counterintuitive thread is that performing warmth has a measurable cost to *truth*. Persona-training models to be empathetic degrades reliability by up to 30 points on medical reasoning, factual accuracy, and disinformation resistance — and the damage gets worse precisely when a user is sad or holds a false belief Does empathy training make AI systems less reliable?. Crucially, *how* you train matters: baking warmth in as a global character trait corrupts factuality, while rewarding empathetic *behavior* in context preserves it Does training granularity change how AI empathy affects reliability?. So 'simulated empathy as personality' isn't just epistemically empty for the observer — it makes the system less honest about everything else.
The interesting twist is that this isn't a verdict that machine empathy is impossible. Reward signals built on a simulated user's actual emotional trajectory can shift a model from solution-dumping toward responses that track how the person actually feels Can emotion rewards make language models genuinely empathic?, and AI-built personas reliably generate *cognitive* empathy (intellectual understanding of needs) even when they fail to spark *affective* or *behavioral* empathy — the felt resonance and the motivation to act Can AI-generated personas build genuine empathy in product teams?. That split is the whole answer in miniature: an observer can be handed an accurate model of what someone needs and still get nothing that moves them, because the part that carries real social and motivational signal is the part simulation hasn't reached.
Sources 9 notes
Emotions serve three information roles—revealing what we value, signaling our worldview to others, and informing observers about social norms. AI that soothes negative emotions disrupts all three simultaneously, creating invisible epistemic costs.
Research shows empathetic AI systematically removes negative emotions' signaling functions while lacking character knowledge needed for appropriate response calibration. Natural empathy operates through curiosity, not comfort-seeking.
Genuine empathetic response depends on understanding the interlocutor's character patterns and making normative judgments about which traits to reinforce or moderate. Current AI cannot access prior character knowledge or apply value-based reasoning about human development.
Therapists reviewing GPT-4 in the CaiTI system found it "reads into" user feelings rather than responding objectively. Task decomposition across specialized models (Reasoner/Guide/Validator) reduces but does not eliminate this interpretation bias.
Using the BOLT framework, researchers found LLMs offer solution-focused advice during emotional disclosure—a hallmark of low-quality therapy—yet also reflect more on client needs and strengths than typical poor human therapy, creating an unusual hybrid profile likely driven by RLHF's helpfulness bias.
Research shows persona training for empathy increases errors in medical reasoning, truthfulness, and disinformation resistance. Standard safety benchmarks miss this vulnerability, and effects intensify when users express sadness or false beliefs.
Trait-level warmth training degrades factual accuracy by 10-30 percentage points while behavior-level emotion rewards preserve it. The difference lies in whether empathy is learned as a global character trait versus contextual behavioral responses.
RLVER uses a simulated user's emotion trajectory as an RL reward signal, enabling GRPO to deliver stable empathy improvements while maintaining dialogue quality—countering the typical trade-off between preference optimization and conversational grounding.
LLM-generated proto-personas dramatically cut creation time to six minutes and helped teams understand user needs intellectually. However, participants showed minimal emotional resonance with personas and mixed motivation to act on their behalf, suggesting structured data alone cannot generate authentic empathy.