Showing an AI's plan before it acts, plus asking clarifying questions, cut wasted back-and-forth in a VR test — but did it just make the result more predictable, not better?
Does pairing visual previews with clarification reduce interaction rounds across domains?
This explores whether showing people a preview of what an AI is about to do, alongside asking them clarifying questions, cuts down on back-and-forth rounds, and whether that holds outside the one setting where it has been tested.
This explores whether showing people a preview of what an AI is about to do, alongside asking them clarifying questions, cuts down on back-and-forth rounds, and whether that holds beyond a single domain. The short answer: the corpus has one direct test, and it's in VR. In a 24-person study of editing 3D shapes by voice, pairing clarification questions with previews placed in the 3D scene meant fewer conversation rounds and steadier progress than having no disambiguation at all Do spatially-anchored previews stabilize VR geometry editing tasks?. The detail worth noticing is that peak performance didn't improve. The pairing made the process more predictable without making the final result better. So the payoff is less wasted effort, not higher quality, which matters if you're deciding whether to build it.
On the 'across domains' part, the corpus has no study that repeats this test in a 2D interface, a text editor, or a shopping agent, and nobody should claim it does. What the corpus does show is why the clarification half of the pairing is likely to matter wherever it's tried. Preference training such as RLHF pushes models to answer confidently instead of asking. It cuts 'grounding acts' (clarifying questions and checks that the model understood) to about 77.5% below human levels. Models end up looking helpful on single turns and quietly failing in longer conversations Does preference optimization harm conversational understanding?. A conversation-analysis framework describes the same gap for tool-using agents. Agents chain tool calls silently and drift away from what the user meant. Classic 'insert-expansions' (short side questions asked mid-task) offer a principled way to decide when an agent should stop and ask When should AI agents ask users instead of just searching?. Seen this way, the VR preview is one concrete form of a grounding act. It shows the user what the system understood before the system commits.
An alternative route avoids asking at all. One system watches the user's screen and infers their goal for the moment. People preferred its output 66–86% of the time over a standard LLM's Can systems infer what users want from watching their screen?. That raises an open design question: in a given domain, is it better to cut rounds by asking well and showing a preview, or by inferring well and skipping the question? The corpus doesn't compare the two directly.
The visual half has a looser parallel in GUI agents. Agent S, an agent that operates software through its interface, did better when it combined screenshots with a structured accessibility tree (the text outline of on-screen elements) than with raw screenshots alone Can structured interfaces help language models control GUIs better?. That result concerns the agent's own understanding, not the user's, but it points to the same idea: visual context helps most when it's tied to something precise. In VR that means a preview placed at an exact spot in the scene. For GUI agents it means elements the agent can address by name. Taken together, the evidence that previews plus clarification reduce rounds is solid in 3D editing and only suggestive elsewhere. The cross-domain study hasn't been done.
Sources 5 notes
In a 24-participant VR study, clarification questions combined with 3D previews significantly reduced task progression variability and required fewer conversation rounds than no disambiguation, though peak task performance remained unchanged across conditions.
RLHF optimizes models for single-turn helpfulness by rewarding confident responses over clarifying questions and understanding checks. This preference alignment systematically reduces grounding acts by 77.5% below human levels, creating an alignment tax where models appear helpful but fail silently in multi-turn contexts.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Systems that passively observe user activity to infer just-in-time objectives produce outputs preferred 66–86% of the time over standard LLMs. This approach sidesteps fixed training objectives by inferring narrow, momentary goals from behavior instead.
Agent S's dual-input design—visual input for environmental understanding plus image-augmented accessibility trees for grounding—achieved 9.37% improvement over baseline by factoring planning and grounding into separate optimization paths rather than forcing end-to-end prediction.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- When Are LLM Inferences Acceptable? User Reactions and Control Preferences for Inferred Personal Information
- Bridging the gulf of envisioning: Cognitive design challenges in llm interfaces.
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent
- Insert-expansions For Tool-enabled Conversational Agents
- Beyond Conversations: Spatially-Anchored Previews for Intent Disambiguation in LLM-Assisted Geometry Editing in Virtual Reality
- Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games
- When Single-User-Oriented LLM-based Assistants Involve Others: A Scoping Review of Pathways, Risks, and Responses