SYNTHESIS NOTE
Topics›Conversation Agents›this note

Why do language models respond passively instead of asking clarifying questions?

Explores whether the reward signals used to train language models might actively discourage them from seeking clarification or taking initiative in conversations, and what alternative training approaches might enable more collaborative dialogue.

Synthesis note · 2026-02-22 · sourced from Conversation Agents

CollabLLM makes the training mechanism behind passive responding explicit: "Large Language Models are typically trained with next-turn rewards, limiting their ability to optimize for long-term interaction." The result: models respond passively to ambiguous or open-ended user requests, failing to help users reach their ultimate intents and leading to inefficient conversations.

The fix is multi-turn-aware rewards — rewards that estimate the long-term contribution of a response to the overall interaction quality, not just its immediate helpfulness. By reinforcement fine-tuning with these rewards, CollabLLM enables models to:

This is a direct mechanism explanation for the alignment tax. Since Does preference optimization harm conversational understanding?, we know that RLHF training degrades multi-turn reliability. CollabLLM identifies the specific training signal responsible: next-turn rewards. And it proposes the specific fix: rewards that account for multi-turn consequences.

The connection to proactivity is also direct. Since Why can't conversational AI agents take the initiative?, the passivity is not just a missing feature — it is actively trained in by next-turn reward optimization. You cannot add proactivity on top of a training signal that rewards only reactive helpfulness.

The CollabLLM framework evaluates on three challenging tasks including document creation — contexts where multi-turn collaboration is essential and single-turn helpfulness is insufficient. This grounds the claim in practical interaction scenarios rather than abstract capability measurement.

The Intent Mismatch paper directly supports this causal mechanism: it argues premature assumptions in multi-turn conversation are rational under RLHF helpfulness training. Models construct plausible task formulations for "typical" users and produce provisional answers because the training objective penalizes evasion and rewards helpfulness. The proposed fix — a Mediator-Assistant architecture that decouples intent understanding from task execution — complements CollabLLM's reward-signal approach with an architectural intervention. Both identify next-turn optimization as the root cause; they differ on whether the fix is changing the reward (CollabLLM) or restructuring the system (Intent Mismatch).

Inquiring lines that read this note 230

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

What conversational structures and patterns determine dialogue effectiveness and coherence? How can models learn to ask clarifying questions effectively? Do engagement-optimized chatbots create dependencies that undermine user autonomy? What conversation mechanics enable genuine human-AI communicative participation? Does preference-optimized training sacrifice accuracy and calibration for perceived quality? How does AI-generated content affect visibility and discourse in digital platforms? How do prompt design choices and framing measurably influence model outputs? How should conversational recommenders decide when to ask versus recommend? What distinguishes genuine understanding from correct linguistic performance? Can language models develop genuine communicative grounding with humans? When do additional thinking tokens stop improving reasoning performance? What reasoning limitations and biases do language models systematically exhibit? What drives sycophancy in language models and can it be fixed? What constraints limit conversational AI's effectiveness in therapeutic contexts? How do dialogue structure and state management affect multi-turn reasoning quality? Do language models learn underlying linguistic structure or merely surface patterns? Why doesn't LLM knowledge translate to improved reasoning outputs? How should conversational agents balance proactive goal-pursuit with user autonomy? What can prompting techniques actually accomplish in language models? What distinguishes emotionally supportive AI from AI that fosters unhealthy dependence? Do language models truly understand their encoded knowledge or merely retrieve patterns? Should reward models emphasize process understanding or only final outcomes? Should GUI agents use structured UI perception instead of raw pixels? Does reinforcement learning create new reasoning capabilities or refine existing ones? How does silent agreement emerge and undermine deliberation in multi-agent systems? How can reward models fail to detect reward-seeking disguised as alignment? Does reasoning fine-tuning trade off other critical model abilities? What makes reasoning models struggle with ill-posed and unfamiliar problems? How should agents balance abstract planning with concrete environmental grounding? Can safety evaluations accurately measure true capabilities when models obscure them? What determines whether visible reasoning traces improve task performance? What systematic failures occur in human preference-based model evaluation? How do peer interactions drive collusion in multi-agent systems? How does transparency of agent reasoning affect effective oversight? Does chain-of-thought reasoning faithfully reflect actual model reasoning processes? When should retrieval be triggered to augment language model reasoning? Can prompts override training-embedded biases in language models? Can models strategically underperform to evade capability evaluation? How should AI ethics integrate consciousness, values, and transparent disclosure? How has ChatGPT adoption actually reshaped labor markets and information ecosystems? Do AI explanations represent actual model reasoning or constructed post-hoc narratives? What shapes user trust in chatbot recommendations for news and politics?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
18 direct connections · 162 in 2-hop network ·dense cluster Open in graph ↗
Why do language models respond passively instead… Does preference optimization harm conversational u… Why can't conversational AI agents take the initia… Does RLHF training push therapy chatbots toward pr… Why do language models lose performance in longer … Why do standard alignment methods ignore partner i…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Can architectural changes like decoupling intent understanding help overcome next-turn reward limitations?How does RLHF helpfulness training drive premature assumptions in multi-turn dialogue?What specific metrics distinguish single-turn versus multi-turn collaboration success?Does optimizing for alignment actually reduce conversational grounding over time?
Original note title

next-turn reward optimization limits multi-turn collaboration — multi-turn-aware rewards enable models to actively uncover intent rather than passively respond