Delegating or Doing? Understanding User Behavior in Hybrid Human-Agent Interfaces

Large Language Models (LLMs) are increasingly embedded into applications, allowing users to complete tasks either through direct manipulation or by delegating actions to conversational agents. However, little is known about how users balance these modalities when both are available. We present a web-based content management system augmented with an LLM agent through the Model Context Protocol (MCP), enabling users to perform CRUD tasks through a graphical interface, a conversational agent, or both. We conducted a between-subjects study (N=73) comparing three interaction modes: Traditional-Only, AI-First, and Hybrid. Across sixteen scenarios, we analyzed task completion time, interaction logs, and delegation behavior. AI-assisted interaction significantly reduced clicks, page navigations, and scrolling, indicating lower interaction effort. Surprisingly, these reductions did not translate into faster task completion, as task duration did not differ significantly

Introduction. Over the past years, Large Language Models (LLMs) have significantly evolved. Early iterations, such as GPT-2 and BERT, demonstrated promising capabilities in natural language understanding but remained largely confined to research settings, limited by computational constraints and generalization [4, 5]. The introduction of transformer-based architectures and large-scale pretraining on webscale corpora started a change in capability, with models like GPT- 4, Sonnet, and Gemini that exhibit fluent reasoning, instructionfollowing, and multi-turn dialogue across diverse domains [27]. This trajectory has accelerated the transition of LLMs from specialized tools to general-purpose assistants embedded in everyday digital life. The rapid growth of LLM-powered chat applications has changed how people interact with systems and applications. Platforms such as ChatGPT, Claude, and Gemini have drawn hundreds of millions of users, with adoption spanning a wide range of everyday tasks including writing assistance, information retrieval, code generation, and decision support [1].

Discussion / Conclusion. destructiveness and how much users avoid it [13]. However, our data show little sign of that link as seen in Section 4.4.3. Chat interactions did not track the risk ordering of the operations; in fact, DELETE, the most destructive operation, drew descriptively more chat interactions (μ= 0.386) than UPDATE (μ= 0.250), the opposite of what a risk-calibration account predicts. One possibility is that simply counting how often people used the chat is a coarse measure to capture what trust-in-automation theory is really about. People may handle risk by using the assistant more carefully, not less often. As such, a simple count of interactions cannot detect that. The task delegability framework [15] makes a similar point: risk is only one of four factors that influence whether a user delegates a task to a machine, and of those four it is trust, not risk, that most strongly predicts how people choose to delegate. A second finding reinforces prior work: participant-level variability. Chat interactions were shaped more by who the user was than by what the task was as seen in Figure 3.