Mapping the Design Space of User Experience for Computer Use Agents

Abstract Large language model (LLM)-based computer use agents execute user commands by interacting with available UI elements, but little is known about how users want to interact with these agents or what design factors matter for their user experience (UX). We conducted a two-phase study to map the UX design space for computer use agents. In Phase 1, we reviewed existing systems to develop a taxonomy of UX considerations, then refined it through interviews with eight UX and AI practitioners. The resulting taxonomy included categories such as user prompts, explainability, user control, and users’ mental models, with corresponding subcategories and example design features. In Phase 2, we ran a Wizard-of-Oz study with 20 participants, where a researcher acted as a web-based

Introduction. Intelligent computer use agents, also referred to as “UI control agents,” have emerged rapidly in the last year [3, 27, 29, 30]. Powered by Large Language Models (LLMs) or Multimodal Large Language Models (MLLMs), these agents execute user commands by interacting with available UI elements, such as by tapping buttons or scrolling pages [35]. While significant research has focused on developing agent models and evaluative benchmarks [1, 17, 21, 35], few have studied how human users might want to interact with and make use of these agents, and even fewer have investigated how user interaction with a computer use agent could be designed or what factors should be considered. To address this gap, we investigate: What is the design space of user experience (UX) for computer use agents? In this paper, we map the design space in a two-phase formative study. In Phase 1, we reviewed existing systems to develop a taxonomy that categorizes different aspects related to UX. We analyzed 9 existing computer use agents that were announced in 2024 and 2025 and developed a set of categories describing the design space. We then conducted an interview study with a diverse group of eight practitioners in UX and AI, in which we presented our initial taxonomy and collected their feedback. This resulted in refinements to the taxonomy, including new wording to describe categories and modifications to the categories and their sub-categories. The final design space includes the following areas: user prompts, explainability of agent activities, user control, and the user’s mental model of computer use agents. These areas are then sub-categorized, as we will describe later in the paper. While Phase 1 focused on building the taxonomy from existing computer use agents and practitioner feedback, Phase 2 sought to further evaluate the taxonomy’s coverage and examine which aspects participants valued across different contexts. To this end, we conducted a Wizard-of-Oz study with 20 participants using a browser-based computer use agent. A researcher acted as the web browser–based computer use agent and carried out common tasks based on user commands while probing for participants’ reactions and needs during execution. Scenarios involving errors and risks were also considered. In phase 2, we validated the coverage of the taxonomy and gained a deeper understanding of end users’ experiences and needs when interacting with computer use agents. In summary, this paper contributes a taxonomy of UX considerations for computer use agents, developed through an analysis of existing systems, iterated with practitioner feedback, and validated through a Wizard-of-Oz study with end users. In addition, we contribute empirical insights into users’ experiences and expectations when interacting with computer use agents, enriching our understanding of the design space by identifying the connections between design areas and divergence in user needs and scenarios. We expect the taxonomy to be useful in helping a designer balance the factors and features that arise when designing a new computer use agent. We also invite future work to identify use cases of computer use agents that are not included in this taxonomy, and to contribute new dimensions that extend and refine it.

Related work. The idea of computer use agents—software systems that perform tasks through interacting with UIs—has deep roots in the HCI community. For several decades, researchers have explored interface agents that can autonomously change user interface states at a user’s command [23]. Beyond autonomous operation, prior work has also investigated how to combine agent-based control with direct user manipulation, enabling collaboration between users and agents [13, 33]. Early applications of the idea of computer use agents spanned a variety of domains. One notable example was CoScripter [19], a browser plug-in that enabled end users to record and replay actions in a web browser through human-readable scripts. Vegemite extended CoScripter with features specifically designed to support data analysis tasks [25]. ScriptAgent leveraged programming by example to record procedures that could be generalized by the agent [24]. Kozierok and Maes [18] introduced an agent that applied reinforcement learning to adapt to a user’s personal scheduling preferences through observation and feedback. Similarly, Lieberman [23] supported web browsing by conducting a concurrent breadth-first exploration based on the current navigation state. While these early systems laid the foundations for autonomous UI control, recent advances in LLMs and MLLMs have enabled a new generation of computer use agents—autonomous systems capable of understanding, navigating, and interacting with digital interfaces [35]. Examples of these systems include ILuvUI [17] and Spotlight [20], which focus on single-screen UI tasks such as screen summarization and widget interaction. Ferret-UI [41] and its successor Ferret-UI 2 [21] extend these capabilities with grounding and reasoning for UI interaction. Other efforts have targeted web environments, such as WebVoyager [12], OS-Copilot [40], CowPilot [14], and OpenWebAgent [16], with features ranging from autonomous browsing to user-supervised execution. AutoDroid [39] combines commonsense knowledge from LLMs with domain-specific knowledge of Android apps to enable general task automation, and AXNav [34] applies LLM agents to natural language–based accessibility testing. While much of the current research has focused on model design, benchmarks, and evaluation techniques [1], little attention has been paid to the user experience of interacting with computer use agents. Yet, prior work on pre-(M)LLM systems has shown that user experience is a critical dimension. Shneiderman and Maes [33] argued that “agents are not alternatives to direct manipulation,” while Horvitz [13] proposed that mixed-initiative systems must carefully balance agent autonomy with user control.

Method. 3 Phase 1: Building the taxonomy of UX for computer use agents 3.1 Methods 3.1.1 Developing the initial taxonomy based on existing computer use agents. We started the development of the taxonomy by reviewing 9 computer use agents that were announced to the public in 2024 to 2025 as shown in Table 1. These agents represent the state of the art in UI control and operate across a range of platforms, including desktop, browser, and mobile. For each agent, the research team reviewed the demo videos if available, and installed and used the agent for 30 minutes for the ones that were available. We took field notes during the demo videos and usage sessions, and discussed as a team the significant themes that we observed. As a result of those discussions, we came up with an initial taxonomy containing a categorization of the design space for LLMbased computer use agents. The taxonomy contains themes and sub-themes of design considerations related to user experience, specifically: user prompts, explainability of agent activities, user control, end of task indicators, and task preview. For each of the themes, we also included examples of design features sourced from the set of computer use agents that we reviewed. We also included speculative design features that the research team imagined but did not see among the existing computer use agents.

3.1.2 Iterating the taxonomy. We then presented our initial version of the taxonomy in an interview study with 8 practitioners who are designers, engineers, or researchers working in the domains of UX or AI at a large technology company. We recruited the participants through internal email lists, message channels, and snowball sampling. Our study protocol with participants were reviewed and approved by a research ethics committee for human subjects and legal counsel at our company. The goal of the study was to collect feedback on our initial taxonomy, specifically how each theme aligns with design considerations, whether there are any important omissions, and how it could be extended with sub-themes (e.g., for the theme about agent mistakes, what are the different types of agent mistakes?).

3.2 The taxonomy of UX for computer use agents Table 2 presents the complete taxonomy, which follows a three-level hierarchy including category, the high-level areas of UX considerations in the design space; subcategory, the specific aspects to consider in a particular area; and example features, illustrative features supporting the UX consideration in a subcategory that are observed in existing agents or speculated by the research team. The taxonomy covers four categories of UX considerations in the design space: User query, Explainability of agent activities, User control, and Mental model. Each category, subcategory, and the corresponding example features are explained below.

3.2.1 User Query. This category covers the design considerations involved in supporting users as they input commands to the agent. It includes the following subcategories: The design of the user experience for issuing commands to an agent should account for the varying levels of expression users may employ in their query. For instance, a user might directly speak the gesture by stating the action and the UI element (e.g., “Click the back button”). Alternatively, the user might express their intent in a more semantic way (e.g., “I want to go back to the last page”). It is also important to consider when the user enters a query during their interaction with the agent. In some cases, the user may provide a single query at the beginning and expect the agent to carry out the task autonomously. Alternatively, the user may anticipate a more conversational interaction, entering follow-up queries for corrections or confirmations and taking turns with the agent. The modality of user input is another important factor to consider. Input modalities may include text, images, voice, and others, each of which can provide cues about the user’s goals and the context of the interaction.

Discussion. While much of the current research has focused on model design and evaluation benchmarks, our paper is the first to examine the space of user experience with computer use agents in depth. We contribute a taxonomy that identifies four major areas shaping the user experience of computer use agents: user query, explainability of agent activities, user control, and user mental models & expectations. Within these areas, we propose 21 dimensions (i.e., subcategories) that developers can consider when designing user experiences for computer use agents. We validated this taxonomy through a Wizard-of-Oz study and further described the divergent end-user scenarios and needs regarding each design areas. Notably, the design dimensions listed in the taxonomy are explicitly non-opinionated. We do not intend for this taxonomy to provide guidance on whether a particular design is good or bad, nor on what should be prioritized over other considerations. We believe the field of computer use agents is too nascent, rapidly evolving, and insufficiently mature to confidently offer prescriptive guidelines at this time. As shown in the Phase 2 study, aspects of design captured by the taxonomy may be essential for one use case while offering little or no benefit in another, and thus require additional contextdependent investigation before any definitive recommendations can be made. Furthermore, we do not claim that this taxonomy is completely exhaustive. As discussed in Section 5.1, there are many directions along which it may be extended. We invite future work to identify use cases of computer use agents that are not included in this taxonomy, and to contribute new dimensions that extend and refine it. In the following sections, we discuss some key takeaways from our studies by connecting our findings with design considerations about pre-LLM era interface agents and general human-AI interaction systems. We invite developers to use our taxonomy as a map and a list of vocabulary for what to consider about user experience in the design of computer use agents, and to reflect on, experiment with, and account for the potential trade-offs regarding the diverse user needs and scenarios.

Handling ambiguity about UI elements in user queries: As Horvitz [13] suggested in the critical factors of mixed initiative interface agents, agents should handle ambiguity by “considering uncertainty about a user’s goals”, “inferring ideal action in light of costs, benefits, and uncertainties”, and “employing dialog to resolve key uncertainties.” Our findings from both the taxonomy development and the Wizard-of-Oz study echo this early vision, and we extend it by providing details in the context of (M)LLM-based computer use agents. Ambiguous queries can occur in cases where a single query may refer to interactions with multiple UI elements on the screen, and such ambiguity may stem from different levels of user expression (e.g., expressing intent rather than gestures) or from the user’s goals (e.g., during exploratory UI tasks). When designing computer use agents, it is critical to consider how and when the agent should make decisions on which UI element to interact with when the user query is ambiguous, and how the agent should adapt to different contexts and user goals. For example, users may prefer the agent to make reasonable assumptions automatically when the risk of taking a diverting UI path is low, such as navigating to a webpage section through search instead of through navigation menu. However, in high-stakes or impactful situations, such as those involving payments, users do not want the agent to act automatically and instead prefer to be informed of possible options and have the agent request clarification before proceeding with an action.

Conclusion. In this two-phase formative study, we explore the design space of user experience for computer use agents. In Phase 1, we developed a taxonomy that maps the general design space of UX for such agents, drawing from an analysis of nine recently announced agents and interviews with eight practitioners in UX and AI. The taxonomy highlights key areas of design consideration, including: what to account for when supporting user queries; which aspects of the agent’s activities should be explained to the user; how to grant users appropriate control over different parts of the interaction; and what constitutes the user’s mental model of the agent. In the Phase 2, we conducted a Wizard-of-Oz study with 20 participants, where a researcher acted as a web browser–based computer use agent to carry out tasks related to vacation rentals and online purchases based on user commands, probing for user reactions and needs both during normal execution and in situations involving errors or risks. The findings validate the taxonomy from Phase 1 and enrich the design space with deeper insights into user experience. Through this two-phase approach, we contribute both a taxonomy and empirical findings on UX for computer use agents. Future designers and developers can refer to this taxonomy to systematically account for different aspects of user experience in the design of computer use agents. We also invite future work to identify additional use cases of computer use agents and to extend this taxonomy with new dimensions as the technology evolves.

Limitations. 5.1 Limitations and Future Work This work is qualitative and exploratory by its nature, and therefore has several limitations. Although the taxonomy developed in Phase 1 is intended to be platform-agnostic, Phase 2 focused exclusively on Wizard-of-Oz scenarios for browser-based tasks, which limits insights about its applicability to other platforms. In both phases, all participants were recruited internally from our company, which may introduce demographic and professional bias toward tech-savvy, well-educated, English-speaking populations, thereby limiting external generalizability. As a result, some important applications of computer use agents, such as accessibility, are not examined in this study. Furthermore, this internal recruitment method, driven by company policy, may also lead to bias and responses influenced by social desirability. In Phase 2, because the agent is simulated, certain user experience considerations related to real-world model performance, latency, or multimodal perception may not fully manifest. Recognizing these limitations, this formative work opens up many directions for future research, which we discuss below.