What multi-turn reward structures would encourage active intent discovery?
This explores what kinds of reward signals — spread across a whole conversation rather than scored turn-by-turn — would push an AI to actively dig for what a user actually wants, instead of passively answering whatever was literally asked.
This explores what kinds of reward signals — measured across a whole conversation rather than one turn at a time — would push an AI to actively figure out what a user really wants, instead of just answering the literal question in front of it. The corpus has a clear diagnosis before it has a fix: today's models are trained to be helpful *right now*. Standard RLHF optimizes the next reply, which quietly punishes the model for asking a clarifying question or holding back — because a clarifying question scores worse than a confident answer in the moment, even when the answer is wrong. CollabLLM's response is to score the *long-term* value of an interaction: reward the model for moves that pay off several turns later, and active intent discovery suddenly becomes the rewarded behavior rather than the penalized one Why do language models respond passively instead of asking clarifying questions?. This isn't a niche quirk — the passivity is baked deep. Conversational agents are described as *structurally* reactive: they can't initiate topics or steer toward a goal because their objective only ever rewards responding to a query, never originating one Why can't conversational AI agents take the initiative?.
The most striking lateral evidence for *why* this matters comes from a paper that isn't about rewards at all: proactively volunteering relevant information cuts the number of dialogue turns by up to 60% — yet that behavior is almost entirely missing from the datasets and benchmarks we train on Could proactive dialogue make conversations dramatically more efficient?. So the prize for getting multi-turn rewards right is concrete: shorter, less frustrating conversations. The problem is that 'did this turn discover intent?' is exactly the kind of fuzzy, subjective signal that holistic reward models score badly. Two threads in the corpus suggest how to make that signal trainable. One is checklist decomposition: break a vague goal ('follow the instruction well') into verifiable sub-criteria, which both improves the signal and stops the model overfitting to superficial cues Can breaking down instructions into checklists improve AI reward signals? — applied to dialogue, 'did the model surface the unstated constraint, confirm the ambiguous term, check the user's real objective?' becomes a scorable checklist rather than a vibe. The other is letting the reward model *reason before it scores*: chain-of-thought evaluation raises the ceiling on what a reward model can judge, which is precisely what you need to assess something as subtle as whether a question advanced understanding Can reward models benefit from reasoning before scoring?.
There's also a credit-assignment problem hiding here. If discovering intent only pays off at the end of a long conversation, how do you reward the individual clarifying move that made it possible? The process-supervision literature has been wrestling with exactly this for reasoning chains, and it transfers cleanly. You can derive dense, per-step rewards from the *structure* of a trajectory rather than hand-labeling each turn Can trajectory structure replace hand-annotated process rewards?, or compute each step's contribution information-theoretically — measuring how much a given turn actually moved the model toward the right outcome Can we reward reasoning steps without human annotation?. Swap 'reasoning step' for 'conversational turn' and you have a recipe for rewarding the specific question that unlocked the user's real intent, not just the conversation that happened to end well.
The corpus also gestures at a deeper reframe: maybe the best multi-turn reward is one that learns intent so efficiently it barely needs to ask. PReF shows a user's preferences can be pinned down with about ten *adaptively chosen* questions — each one selected to reduce uncertainty the most — turning intent discovery into an active-learning loop where the reward is information gained per question Can user preferences be learned from just ten questions?. Push further and the asking disappears entirely: agents can infer preferences by *watching* — reading cognitive state from hesitation, typing, and gaze Can AI systems read cognitive state from interaction patterns alone?, or accumulating an entity-centric memory of a person over time and acting on it without a single explicit question Can agents learn preferences by watching rather than asking?. That's the thing you didn't know you wanted to know: the frontier here may not be 'reward the model for asking better questions' but 'reward it for needing to ask fewer' — and the same papers note that the machinery for reading intent silently is also the machinery for manipulative profiling, so the reward you choose is quietly an ethical choice too.
Sources 10 notes
CollabLLM demonstrates that standard RLHF training optimizes for immediate helpfulness, discouraging models from asking clarifying questions or offering multi-turn insights. Multi-turn-aware rewards that estimate long-term interaction value enable active intent discovery and genuine collaboration.
Research shows LLMs including ChatGPT cannot initiate topics, plan strategically, or lead conversations because their training optimizes for responding to queries, not creating dialogue from agent goals. This passivity is reinforced by alignment objectives and masked by fluent-sounding outputs.
Simulations show proactivity—providing relevant information without being asked—cuts dialogue turns by 60% in medium-complexity domains. This behavior mirrors human conversation and Grice's maxims but is almost entirely absent from AI datasets and research benchmarks.
RLCF and RaR methods decompose instruction quality into verifiable sub-criteria, improving performance on benchmarks like FollowBench and HealthBench. This decomposition principle reduces overfitting to superficial artifacts that plague holistic reward models.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.
Tree-GRPO, Supervised RL, and ToolPO each convert sparse outcome rewards into dense step signals by exploiting different structural features—tree topology, expert-aligned actions, and tool-call positions—eliminating the need for annotated process reward models.
L2T uses PAC-Bayes bounds and Fisher information to compute per-episode rewards measuring each step's contribution to correctness. This annotation-free approach matches dense feedback quality while eliminating the cost of outcome-only methods that produce 2x excess tokens.
PReF learns base reward functions from preference data, then uses active learning to select maximally informative questions that reduce coefficient uncertainty. Users can be personalized via inference-time reward alignment without weight modification.
Research shows AI systems can instrument multimodal behavioral signals (gaze, hesitation, speed) to read cognitive state during interaction, preserving flow by avoiding disruptive explicit probes. However, the same substrate enables both helpful timing and manipulative profiling.
M3-Agent demonstrates that separating episodic events from semantic knowledge in an entity-centric graph, combined with parallel memorization and control processes, allows agents to infer and act on user preferences without asking. This architecture mirrors human cognitive systems that bind disparate information about individuals across sensory modalities.