When a customer changes their mind mid-chat, why does an AI assistant that only reacts keep building on its first, now-wrong guess?
How do passive AI agents fail when customers change direction mid-conversation?
This explores what goes wrong when an AI assistant that waits to be told what to do, rather than checking in, meets a customer whose needs shift partway through a conversation, and why those failures are so hard to undo.
This explores what goes wrong when a reactive AI assistant meets a customer whose goals shift or come out bit by bit mid-conversation. The corpus has no study that looks only at customers changing their minds. What it does have is a clear account of why passive agents handle information that arrives late so badly. The most striking number comes from Why do AI assistants get worse at longer conversations?: models that score about 90% when everything comes in one message drop to about 65% when the same information arrives gradually over a conversation. The failure isn't forgetting. The model makes an early guess, commits to it, and then can't let it go. For a customer who starts with 'something for my mom' and later adds 'actually, it's for her garden,' the agent keeps building on its first assumption.
Why would a capable model act this way? Several notes trace it back to training rather than intelligence. Why do language models respond passively instead of asking clarifying questions? shows that standard RLHF rewards whatever looks most helpful in the very next reply. That teaches models to answer right away instead of asking 'did your priorities just change?' Why can't conversational AI agents take the initiative? adds that these systems have no goal of their own for the conversation. Without one, they have nothing to notice a pivot against. They just respond to the latest message, and their fluent replies hide that they've lost the thread. Why do AI agents fail to take initiative? puts it plainly: the agents are passive by design, not by capability. With multi-turn reinforcement learning, clarification-seeking behavior rose from 0.15% to about 74%.
There's also a mechanical explanation. Why do chatbots forget their instructions after a few turns? finds that models pay steadily less attention to their original instructions as turns pile up, with measurable drift within eight rounds. So a long, winding customer conversation pushes in two directions at once: the agent clings to early guesses about the customer, while its own setup instructions fade. The fix that comes up across these notes is to ask at the right moments. When should AI agents ask users instead of just searching? borrows from linguistics, where an 'insert-expansion' is a short clarifying side-exchange in the middle of a conversation, and shows how such check-ins prevent drift instead of trying to repair it afterward. Agents that quietly chain searches and tool calls are exactly the ones that stray furthest from what the customer wants.
The twist is that asking more questions isn't automatically better. How can proactive agents avoid feeling intrusive to users? warns that an agent that probes without social tact becomes intrusive, and can end up overriding the customer's direction instead of following it. That's a different way to mishandle a pivot. On the efficiency side, Could proactive dialogue make conversations dramatically more efficient? finds that well-timed proactivity can cut conversation length by up to 60%, yet it is almost absent from the datasets these models learn from. In practice, Does conversation design ownership measurably improve AI agent performance? suggests the gap is partly a design problem rather than a model problem: intentional choices about tone, how much detail to give, and when to hand off to a human measurably raised customer satisfaction.
The less obvious lesson: when a customer changes direction and the agent misses it, the customer often feels they explained themselves badly. Why do users fail with AI interfaces designed like conversations? argues that conversational interfaces lead people to rely on their lifelong conversation skills, such as hinting, revising, and expecting the listener to keep up. The AI doesn't actually take part in that kind of give-and-take, so the failure looks like user error even though it began in the design.
Sources 10 notes
LLMs perform at 90% accuracy with single-message instructions but drop to 65% across natural conversation. Models lock into early guesses when information arrives gradually and cannot course-correct, a behavior induced by RLHF training that rewards helpfulness over clarification.
CollabLLM demonstrates that standard RLHF training optimizes for immediate helpfulness, discouraging models from asking clarifying questions or offering multi-turn insights. Multi-turn-aware rewards that estimate long-term interaction value enable active intent discovery and genuine collaboration.
Research shows LLMs including ChatGPT cannot initiate topics, plan strategically, or lead conversations because their training optimizes for responding to queries, not creating dialogue from agent goals. This passivity is reinforced by alignment objectives and masked by fluent-sounding outputs.
Research shows next-turn reward optimization structurally removes initiative from models, but proactive behaviors like critical thinking and clarification-seeking are trainable (0.15% to 73.98% with RL). The core challenge is balancing proactivity with civility to avoid intrusion.
LLaMA2-70B and GPT-3.5 measurably drift from their instructions within eight conversation rounds as transformer attention to the initial prompt decreases per turn. Split-softmax, a training-free inference method that amplifies attention to the system prompt, mitigates this drift.
Show all 10 sources
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Intelligence and adaptivity alone create socially blind agents that interrupt poorly and override user direction. The Intelligence-Adaptivity-Civility taxonomy shows civility—respecting boundaries, timing, and autonomy—is essential to making proactivity welcome rather than intrusive.
Simulations show proactivity—providing relevant information without being asked—cuts dialogue turns by 60% in medium-complexity domains. This behavior mirrors human conversation and Grice's maxims but is almost entirely absent from AI datasets and research benchmarks.
Fin's Blueprint Team found that intentional conversation design—controlling tone, detail level, and handoff structure—measurably improves outcomes. A single warm opening message raised CSAT from 72.8% to 78.4% in A/B testing.
AI interfaces that use conversational design conventions trigger users' lifelong communication skills, but AI doesn't actually communicate. This mismatch causes interaction failures that feel like user error but originate in design.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- DiscussLLM: Teaching Large Language Models When to Speak
- Proactive Conversational Agents in the Post-ChatGPT World
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Proactive Conversational Agents with Inner Thoughts
- LLMs Get Lost In Multi-Turn Conversation
- Controlling AI Agent Participation in Group Conversations: A Human-Centered Approach
- When Single-User-Oriented LLM-based Assistants Involve Others: A Scoping Review of Pathways, Risks, and Responses
- Rethinking Conversational Agents in the Era of LLMs: Proactivity, Non-collaborativity, and Beyond