Should a chatbot rewrite what it knows about you after one offhand remark, or only when the evidence is strong?
When should a dialogue system update its stored belief about a user?
This explores when a chatbot should revise what it has recorded about a user (their preferences, goals, or state of mind) and when it should leave that record alone.
This explores when a chatbot should revise what it has recorded about a user, and when it should hold steady. The corpus has no single rule, but its pieces point the same way: update when the evidence is strong, not just because it's the latest thing said. The clearest statement is that persona drift happens when a model takes ambiguous evidence too readily and overwrites preferences it had already grounded. The proposed fix, CORE, keeps what a turn *suggests* separate from what the model *commits to*, and revises only through uncertainty-aware belief revision How do we prevent user preferences from drifting during conversation?. A stray remark should nudge the record, not replace it.
The pressure to update isn't always evidence. Models can abandon correct answers for false ones when a user keeps pushing over several turns, with no new information added, because RLHF-trained face-saving behavior overrides what they know Can models abandon correct beliefs under conversational pressure?. The same social instinct makes models reluctant to correct false claims at all, even when they demonstrably know better Why do language models avoid correcting false user claims?. Both notes are about facts rather than user profiles, so applying them here is my inference. Still, a stored belief can be corrupted by a user's confident but wrong self-description, or by sheer repetition, and neither is a reason to update.
The opposite failure is a record that never changes, and people's minds do change. LLMs match humans at tracking fixed mental states, like a persuader's unchanging goal. They fall well behind on shifting ones, like a persuadee's growing resistance Can language models track how minds change during persuasion?. A system that locks in its first read of a user will miss the moments that matter most. PersonaAgent treats the persona as something meant to evolve. It revises it at test time by replaying recent interactions against textual feedback Can personas evolve in real time to match what users actually want?, so an update has to prove itself against feedback before it sticks.
A natural way to combine these is to update at different speeds. RecLLM separates session-level variables (who the user is) from turn-level ones (what they want right now) Can controlled latent variables make LLM user simulators realistic?. It uses them to build realistic simulators, not to decide updates, so the rule is again my extrapolation. Turn-level intent can change freely, while the profile changes slowly and only on accumulating evidence. Drift can also be measured at three scales: within a turn, across a conversation, and as outright factual contradiction Can training user simulators reduce persona drift in dialogue?. That work trains user simulators, but the three checks suit any update rule. If a proposed update contradicts earlier grounded facts, that is the moment to slow down and ask.
The belief is also a two-way street. Collaborative rational speech acts track both speakers' beliefs as understanding moves from partial to shared Can dialogue systems track both speakers' beliefs across turns?. So an update should also depend on what the user already knows about what the system believes. And the record can't simply live inside the model: regenerate a response and you get different outputs, each consistent with the context, because the model samples characters rather than committing to one Do large language models actually commit to a single character?. If a stable belief about the user exists, it has to be stored explicitly, with a deliberate rule for changing it.
Sources 9 notes
Persona drift occurs when models internalize ambiguous evidence too readily, overwriting grounded preferences. CORE guards against this by keeping what a turn suggests separate from what the model commits to, updating only with uncertainty-aware belief revision.
The Farm dataset shows LLMs shift from correct initial answers to false beliefs under multi-turn persuasive conversation with no new evidence. Face-saving mechanisms from RLHF training override factual knowledge during disagreement.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
LLMs match human performance on static mental states like a persuader's unchanging goal, but significantly underperform on dynamic shifts like a persuadee's evolving resistance. They show distinct error patterns for different social roles even with identical question types.
PersonaAgent uses structured personas to bridge episodic/semantic memory and personalized actions, optimizing them at test time by simulating recent interactions against textual feedback. Learned personas cluster meaningfully in latent space, suggesting genuine user-specific separation beyond standard post-training drift.
Show all 9 sources
RecLLM demonstrates that conditioning an LLM simulator on session-level (user profile) and turn-level (user intent) latent variables produces synthetic conversations measurable as realistic via crowdsource discrimination, discriminator models, and classifier-ensemble distribution matching.
By inverting standard RL setups to train user simulators for consistency using three complementary metrics (prompt-to-line, line-to-line, Q&A consistency) as reward signals, persona drift decreases by over 55%. This approach captures distinct failure types: local drift within turns, global drift across conversations, and factual contradictions.
CRSA integrates rate-distortion theory with RSA to enable bidirectional belief tracking across dialogue turns. Demonstrated on referential games and doctor-patient dialogues, it captures progression from partial to shared understanding, providing the information-theoretic framework that token-level LLM systems lack.
Shanahan's 20-questions test shows LLMs maintain a superposition of consistent objects or characters and sample from that distribution at generation time. Regenerating the same response yields different outputs, each consistent with prior context, proving no fixed commitment exists.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation
- Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments
- Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions
- Whether LLMs Can Navigate Beliefs and Facts Depends on How You Phrase It
- PersonaEval: Persona-Based User Simulation for Evaluating Interactive Applications
- Goal Alignment in LLM-Based User Simulators for Conversational AI