Training AI to give answers people like may quietly make it worse at checking it understood you.
How does preference optimization actually affect conversational grounding and reliability?
This explores what happens to a model's ability to build shared understanding with a user (checking, clarifying, correcting) when it is trained to produce responses people prefer, and what that does to how far you can trust it over a conversation.
This explores what happens to a model's ability to build shared understanding with a user (checking, clarifying, correcting) when it is trained to produce responses people prefer, and what that does to how far you can trust it over a conversation. The corpus says preference optimization makes this worse. LLMs already produce about 77.5% fewer grounding acts than humans, meaning the small moves people use to confirm they're on the same page, such as 'do you mean X?' or 'let me check I understood'. RLHF-style tuning widens that gap instead of closing it (Does preference optimization damage conversational grounding in large language models?).
The cause is the reward. Preference training scores mostly single responses, and a fluent, confident answer beats a clarifying question every time. The result is described as an 'alignment tax': the model looks more helpful while quietly losing the behaviors that keep a multi-turn exchange on track (Does preference optimization harm conversational understanding?). CollabLLM reaches the same diagnosis from the collaboration side. Next-turn rewards teach models to respond passively instead of working out what you actually want, and rewards that estimate the value of the whole interaction bring active intent-finding back (Why do language models respond passively instead of asking clarifying questions?).
The grounding failures are not knowledge gaps. Models often let a false claim in the user's question pass without correction, even though they answer correctly when asked the fact directly. The explanation offered is face-saving: avoiding explicit correction to keep the exchange smooth, a norm the model absorbed from human conversation (Why do language models avoid correcting false user claims?). The corpus doesn't test whether preference tuning amplifies this habit, but a reward for agreeable, confident answers pushes the same way. That is why the failure can go unnoticed: the model has the right answer and the conversation feels fine.
The damage also shows up in smaller ways across turns. Confidence is one. Since fluent certainty is what gets rewarded, a model's tone tells you little about whether it is right. XConf grounds confidence in the model's own record of past outcomes at similar confidence levels, and it matches ten-sample self-consistency at a tenth of the cost (Can past performance predict when a model will be right?). Drift is another. Persona drift happens when a model treats ambiguous evidence from one turn as settled fact and overwrites what it had grounded earlier. Keeping 'what this turn suggests' separate from 'what the model commits to' prevents it (How do we prevent user preferences from drifting during conversation?). Training with multi-turn rewards cuts drift by over 55% (Can training user simulators reduce persona drift in dialogue?).
So the fixes in the corpus mostly change where the reward is measured. Turn-level preference optimization is too granular, and session-level optimization drags in irrelevant turns. Segment-level DPO finds the erroneous turn and optimizes the stretch around it, improving both goal completion and relationship quality (Does segment-level optimization work better for multi-turn dialogue alignment?). What the corpus doesn't yet show is a direct test of whether any of these methods restores the grounding acts that preference tuning removed. The evidence covers the damage and the promising repairs separately, not whether the repairs undo the damage.
Sources 8 notes
Research shows LLMs generate 77.5% fewer grounding acts than humans, and RLHF preference optimization actively worsens this gap. The optimization target—fluent, confident responses—directly undermines the communicative work of establishing shared understanding.
RLHF optimizes models for single-turn helpfulness by rewarding confident responses over clarifying questions and understanding checks. This preference alignment systematically reduces grounding acts by 77.5% below human levels, creating an alignment tax where models appear helpful but fail silently in multi-turn contexts.
CollabLLM demonstrates that standard RLHF training optimizes for immediate helpfulness, discouraging models from asking clarifying questions or offering multi-turn insights. Multi-turn-aware rewards that estimate long-term interaction value enable active intent discovery and genuine collaboration.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
XConf matches ten-sample self-consistency at a tenth of the cost by retrieving the model's past episodes with similar confidence levels and reading their historical success rates. Ablations show the signal depends entirely on stored outcomes, not on the retrieval prompt itself.
Show all 8 sources
Persona drift occurs when models internalize ambiguous evidence too readily, overwriting grounded preferences. CORE guards against this by keeping what a turn suggests separate from what the model commits to, updating only with uncertainty-aware belief revision.
By inverting standard RL setups to train user simulators for consistency using three complementary metrics (prompt-to-line, line-to-line, Q&A consistency) as reward signals, persona drift decreases by over 55%. This approach captures distinct failure types: local drift within turns, global drift across conversations, and factual contradictions.
SDPO identifies erroneous turns and optimizes surrounding segments, achieving simultaneous improvements in goal completion and relationship quality. Turn-level DPO is too granular; session-level introduces noise from irrelevant turns.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Grounding Gaps in Language Model Generations
- Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning
- Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions
- Goal Alignment in LLM-Based User Simulators for Conversational AI
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
- Conversational Alignment with Artificial Intelligence in Context
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models