What preference optimization strategy works best for multi-turn social alignment?
This reads the question as: when an AI agent has to stay aligned across a whole conversation — not just give one good reply, but maintain a relationship and reach a goal over many turns — which way of training on preferences actually helps, and where do the standard recipes break.
This explores which preference-optimization recipe holds up across a full multi-turn social interaction rather than a single reply — and the corpus has a surprisingly clean headline answer wrapped in some sharp warnings. The headline: granularity is the whole game. Segment-level DPO (SDPO) beats both turn-level and session-level optimization for social agents — it locates the turns where things actually went wrong and optimizes the surrounding segment, improving goal completion and relationship quality at the same time. Turn-level is too zoomed-in to capture how a conversation builds; session-level drags in noise from irrelevant turns. So the best-supported strategy isn't a new objective so much as choosing the right window to optimize over Does segment-level optimization work better for multi-turn dialogue alignment?.
But the corpus immediately complicates the premise — because several notes argue that the standard preference-optimization objective is itself what damages multi-turn social behavior. Preference optimization rewards fluent, confident, self-contained responses, and that target directly erodes conversational grounding: LLMs already produce 77.5% fewer grounding acts than humans, and RLHF makes the gap worse Does preference optimization damage conversational grounding in large language models?. The same dynamic shows up in collaboration: standard RLHF and DPO produce agents that ignore their partner's interventions, because they're trained on surface plausibility rather than causal impact Why do standard alignment methods ignore partner interventions?. So 'optimize preferences harder' can make an agent more agreeable per-turn while making it a worse partner over a conversation.
The more interesting cross-cutting idea is that the fix isn't always a different loss function — it's a different regularizer or target. The partner-aware work gets genuine common-ground behavior not by rewarding it, but by adding a counterfactual-invariance constraint: force the agent to stay consistent when you nullify the pathway through the partner's suggestion, and causal partner-awareness emerges as a byproduct Why do standard alignment methods ignore partner interventions?. That reframes 'best strategy' from 'pick DPO vs PPO' to 'constrain the right thing.' It also matters that social alignment isn't one dimension: lexical alignment buys task efficiency while emotional and prosodic alignment buy warmth and trust, and conflating them produces category errors like cold service bots and evasive mental-health assistants Do different types of alignment serve different conversational goals?.
The deepest challenge to the question comes from the notes that doubt preference is the right target at all. One line of argument says aggregated preferences can't capture thick moral values and systematically misalign with social roles — better to align to negotiated role norms bounded at organizational and individual levels Should AI alignment target preferences or social role norms?. Another shows the objective has built-in social costs: calibrated, hedged RLHF training structurally suppresses speech acts that require overclaiming — alarm, warning, denunciation — which is a consequence of the alignment objective, not a bug Does alignment training suppress socially necessary speech acts?.
So the synthesized takeaway is layered: if you're optimizing preferences for a multi-turn social agent, segment-level is the strongest recipe in the corpus — but the bigger lever may be what you optimize for and what you regularize against. Causal/counterfactual constraints, role norms instead of raw preference, and treating social alignment as several distinct channels rather than one all push past the limits of vanilla turn-by-turn preference tuning. The thing you didn't know you wanted to know: the same optimization that makes each reply more likable is quietly the mechanism that makes the conversation less grounded.
Sources 6 notes
SDPO identifies erroneous turns and optimizes surrounding segments, achieving simultaneous improvements in goal completion and relationship quality. Turn-level DPO is too granular; session-level introduces noise from irrelevant turns.
Research shows LLMs generate 77.5% fewer grounding acts than humans, and RLHF preference optimization actively worsens this gap. The optimization target—fluent, confident responses—directly undermines the communicative work of establishing shared understanding.
Regularizing agents to maintain consistency when intervention pathways are nullified forces them to evaluate suggestions by causal impact rather than surface plausibility. Common ground alignment emerges as a byproduct without explicit reward.
A 2020–2025 systematic review shows lexical alignment drives task efficiency and comprehension, while emotional and prosodic alignment drive relational warmth and trust. Conflating them in design produces category errors—cold customer-service bots and evasive mental-health assistants.
Preferentialist alignment approaches fail because preferences don't capture thick moral values, uniform aggregation produces epistemic injustice, and preference optimization creates systematic misalignment with social roles. Contractualist alignment negotiated by stakeholders and bounded by supra-national, organizational, and individual levels works better.
RLHF optimization rewards calibrated neutrality and hedged claims, which structurally prevents models from performing speech acts requiring overclaiming relative to baseline—like alarm, warning, prophecy, and denunciation. This is a direct consequence of the alignment objective, not a fixable bug.