Should an AI agent act differently when it's chatting by voice versus driving your screen by clicks?
How should AI agents adapt their interaction style across different devices and modalities?
This explores whether AI agents should talk and act differently depending on where they show up (a chat window, a voice assistant, a phone, a browser they're driving), and what the collection says about tailoring interaction to the surface and the person.
This explores whether AI agents should change how they interact depending on the device or medium they appear in: chat, voice, screen-driving, embedded workflows. Directly, the collection has little on device-by-device adaptation. You won't find a study comparing how one agent should behave on mobile versus desktop or in voice versus text. What it does have is a more useful reframing. The thing an agent should adapt to is not mainly the device. It's how much the user wants to steer, how much context the medium can carry, and who the interface is really built for.
The most surprising thread is that the interface itself can be designed for the agent as well as for the human. When researchers added agent-specific cues to interfaces, meaning cues that help an agent perceive, locate and operate on-screen elements, agents completed computer tasks more often and more efficiently. Human users noticed no loss in usability Can interface design help agents complete computer tasks better?. So 'adapting to modality' runs in both directions. Screens may increasingly be designed to serve two kinds of user at once. Meanwhile, Friedman argues that chat is often the wrong medium for humans: typing out what you want is slow. Friedman suggests intent-capture widgets, 'style lenses' that reshape output, and canvas-style direct manipulation as alternatives that sit alongside conversation Can task-oriented UI patterns reduce the friction of AI chat interfaces?. The lesson is that the right interaction style may not be conversational at all.
If you're looking for something to adapt to, the clearest signal in the collection is the person, not the device. In 400 real sessions with web agents, users settled into four stable styles: Hands-off, Hands-on, Collaborative and Full Takeover Do users develop stable collaboration styles with web agents?. An agent that reads which style it's dealing with could set how often it checks in. Conversation analysis offers a formal way to decide when to pause and ask rather than silently chain tool calls When should AI agents ask users instead of just searching?. That trade-off is sharper in low-bandwidth media like voice, where every clarifying question costs more. A related tension: agents are passive by default, but asking clarifying questions and taking initiative can be trained in. The hard part is staying proactive without becoming intrusive Why do AI agents fail to take initiative? Why can't conversational AI agents take the initiative?.
There's also a quieter point about why moving between media is hard. AI context (prompt, history, retrieved data) keeps shifting and doesn't persist, unlike a fixed screen layout that users can learn by heart How does AI context differ from conventional software context?. Each new medium shows the user a different slice of that context. That suggests the design work lies in managing context across surfaces more than in reskinning the agent's personality. On the practical side, Fin's team recommends keeping tone and behavior rules in the agent's instructions and factual content in a knowledge base it can look up Should factual content and behavior rules live separately?. That separation is what would let style flex by channel while the facts stay put. Tone is worth getting right: a single warm opening message raised customer satisfaction by more than five points Does conversation design ownership measurably improve AI agent performance?.
The takeaway you might not have expected: the collection suggests agents should adapt to the user's control style and to how much context the medium can carry, and that interfaces may adapt to agents just as much. For true device- and modality-specific guidance (voice, wearables, mobile), the collection is thin. That's a gap, not a settled question.
Sources 9 notes
UI-Verse benchmark testing found that augmented heuristics targeting agent perception, grounding, and control consistently improved agent task completion and efficiency. Human studies confirmed these designs preserved usability for human users with no observable regressions.
Friedman identifies five design levers—intent capture, output shaping, refinement, agentic tasks, and workflow embedding—that sidestep slow typed prompts. Examples include node-based intent visualization, style lenses for output, and canvas manipulation, shifting AI interfaces away from conversational chat toward direct interaction.
Analysis of 400 real-user sessions reveals intervention patterns cluster into Hands-off, Hands-on, Collaborative, and Full Takeover styles. These styles reflect how users balance supervision and control, emerging from repeated error-correction, preference-refinement, and assistance motivations rather than one-off reactions.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Research shows next-turn reward optimization structurally removes initiative from models, but proactive behaviors like critical thinking and clarification-seeking are trainable (0.15% to 73.98% with RL). The core challenge is balancing proactivity with civility to avoid intrusion.
Show all 9 sources
Research shows LLMs including ChatGPT cannot initiate topics, plan strategically, or lead conversations because their training optimizes for responding to queries, not creating dialogue from agent goals. This passivity is reinforced by alignment objectives and masked by fluent-sounding outputs.
AI interactions operate on a substrate of constantly shifting context—prompt, history, retrieved data, hidden state—that users cannot internalize like traditional UIs. This structural mutability demands a new design discipline centered on context engineering rather than interface design.
Fin's Blueprint Team recommends routing factual content to retrievable knowledge bases while keeping only tone and behavioral directives in agent instructions. This split aligns with instruction-tuning research showing models follow what-to-do rules better than what-to-ignore constraints.
Fin's Blueprint Team found that intentional conversation design—controlling tone, detail level, and handoff structure—measurably improves outcomes. A single warm opening message raised CSAT from 72.8% to 78.4% in A/B testing.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- DiscussLLM: Teaching Large Language Models When to Speak
- Proactive Conversational Agents in the Post-ChatGPT World
- Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents
- Modeling Distinct Human Interaction in Web Agents
- Proactive Conversational Agents with Inner Thoughts
- When Single-User-Oriented LLM-based Assistants Involve Others: A Scoping Review of Pathways, Risks, and Responses
- Probing How Users Interact with Turn-Level Design Frictions for AI Chatbots
- Controlling AI Agent Participation in Group Conversations: A Human-Centered Approach