What design areas matter most for computer use agent UX?

The design space for computer use agent UX divides into four categories, the paper reports: "user prompts, explainability of agent activities, user control, and the user's mental model of computer use agents," further broken into "21 dimensions (i.e., subcategories)." The abstract frames these as answering a felt gap — "little is known about how users want to interact with these agents or what design factors matter for their user experience (UX)" even as "significant research has focused on developing agent models and evaluative benchmarks."

The taxonomy was built in two phases. Phase 1 reviewed "9 computer use agents that were announced to the public in 2024 to 2025," with the research team watching demo videos and using each agent "for 30 minutes" where available, then discussing "significant themes" as a team to produce an initial taxonomy of "user prompts, explainability of agent activities, user control, end of task indicators, and task preview." That draft was revised in an interview study with "8 practitioners who are designers, engineers, or researchers working in the domains of UX or AI at a large technology company," recruited through "internal email lists, message channels, and snowball sampling," producing "refinements to the taxonomy, including new wording to describe categories and modifications to the categories and their sub-categories." Phase 2 then tested the taxonomy's coverage in a Wizard-of-Oz study: "a researcher acted as the web browser–based computer use agent and carried out common tasks" — vacation rentals and online purchases — "based on user commands" with "20 participants," probing reactions to normal execution and to "scenarios involving errors and risks."

Against the nearest notes in the library, this taxonomy sits one level more general than the agent-specific frameworks around it. How should AI systems be designed across different interaction phases? is the pre-agentic ancestor this paper extends into computer-use-specific territory — both consolidate scattered design knowledge into a validated category set, but Amershi's guidelines are organized by when in the interaction they apply, while this taxonomy is organized by what aspect of the agent the design choice concerns. What UX principles do workplace users want in AI agents? narrows the same kind of exercise to workplace agents and names user priorities directly, where this taxonomy stays "explicitly non-opinionated" about which category matters most. How should AI interfaces handle the shift from doing to supervising? cuts similar territory along a different axis, by interface layer rather than by UX category. And How can users verify what an AI agent actually plans to do? is an instance of what this taxonomy's explainability and user-control categories are meant to generalize over.

The paper is careful to bound its own claims: the taxonomy is offered as "a map and a list of vocabulary," not "guidance on whether a particular design is good or bad, nor on what should be prioritized," because the authors judge the field "too nascent, rapidly evolving, and insufficiently mature to confidently offer prescriptive guidelines at this time." Phase 2 validation covered only browser-based tasks despite the taxonomy's platform-agnostic intent, all participants in both phases were recruited internally at one company (introducing "bias toward tech-savvy, well-educated, English-speaking populations" and excluding accessibility use cases), and the Wizard-of-Oz design means "user experience considerations related to real-world model performance, latency, or multimodal perception may not fully manifest." The taxonomy should be read as a vocabulary and a checklist for what to consider, not as evidence about which design choices actually improve outcomes.