Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents

Recent advances have enabled general computer-use agents that interpret screens and execute grounded actions from human instructions, yet they still struggle to generalize to unseen and evolving interfaces. While improving agent capability remains important, agent-compatible interface design offers a complementary path by aligning interaction semantics with agents’ prior knowledge. In this paper, we revisit Nielsen’s 10 usability heuristics through the lens of computer-use agents, identifying which principles naturally transfer, where implicit design assumptions create agent-specific failures, and how safe additive augmentations can improve robustness without harming human usability. To evaluate these ideas, we introduce UI-Verse, a suite of controlled environments built around functionally similar interfaces with different applied heuristics. Experiments show that our augmented heuristics consistently improve task completion and modestly improve efficiency, with combined heuristics yielding further gains. Human studies further show that these designs preserve the original interaction workflow without observable usability regressions. Overall, our findings highlight interface design as a practical complementary avenue for improving the reliability and generalization of computer-use agents.

Introduction. As software interfaces become the front door to both everyday life and professional work, general computer-use agents (CUAs) are emerging as a new route to automation: systems that operate software directly, much like humans do (Agashe et al., 2024; Wang et al., 2025; Hu et al., 2025). Powered by large multimodal models, they are able to interpret the screen and take grounded actions to handle tasks like online shopping, web browsing, replying to emails, and even help with daily work (Xie et al., 2024b; Wang et al., 2024; Bonatti et al., 2024a; Pan et al., 2024; Rawles et al., 2025).

However, because real world interfaces are continually evolving and highly dynamic, today’s general CUAs still struggle to generalize to unseen UIs. In practice, they frequently rely on test time scaling and trial and error to navigate to new environments. They behave more like novices than professionals who operate through stable procedural knowledge (Abhyankar et al., 2025; Liu et al., 2026). This limitation is typically treated as a model capability problem. Yet generalization in interactive environments is not determined solely by the agent; it is also shaped by how interface states are represented, how actions are exposed, and how consistently interaction semantics are organized.

While improving agent architectures remains essential, we argue that interface design itself plays a critical and under-explored role in enabling reliable automation. Agent compatible interface design can substantially reduce adaptation burden by aligning system states and interaction structures with agents’ prior knowledge, thereby improving robustness and performance without modifying the underlying model. Motivated by this perspective, we revisit Nielsen’s ten usability heuristics (Nielsen & Molich, 1990) for human centered Overall, our contributions are as follows:

• We identify agent-compatible interface design as an important but underexplored challenge for general CUAs, and argue that robust automationdepends not only on agent capability, but also on how interfaces expose states and actions.

• We propose a human-centered, agent-compatible interface design framework with two general design suggestions and four agent-specific heuristics that extend classical usability principles to better support agent perception, grounding, and control.

• We validate this framework through agent evaluations and human studies, showing that these heuristics consistently improve CUA reliability and efficiency while preserving human usability when designed appropriately.

Related work. User-Centric Interface Design. User-centric interface design studies how interfaces help people perceive system state, act, and recover from errors. Foundational work introduced principles such as visibility, consistency, error prevention, and recovery support (Norman, 1983; Nielsen & Molich, 1990; Nielsen, 1994a). Related work on direct manipulation and minimalist design further emphasized continuous visual representations, incremental and reversible actions, and immediately observable effects, making interfaces more predictable and easier to control (Kang & Kim, 2007; Shneiderman, 1997). These ideas remain influential in AI-mediated settings. Human-centered AI emphasizes preserving human agency and helping users interpret and guide intelligent systems (Capel & Brereton, 2023), while later work extends classical design principles to generative AI applications, GUI-based machine learning tools, and AI-assisted design workflows (Weisz et al., 2024; Yamani et al., 2024; Lu et al., 2024). However, these efforts still primarily optimize interfaces for human users and do not systematically address settings where reliable operation by general UI-centered agents is also a core objective. Our work builds on this literature but shifts the focus toward human-centered, agent-compatible interface design.

General Computer-Use Agents. Recent work has substantially advanced general computer-use agents. Benchmarks such as OSWorld, OfficeBench, and Windows Agent Arena evaluate agent performance in realistic environments (Xie et al., 2024a; Wang et al., 2024; Bonatti et al., 2024b). Most prior work, however, focuses on improving the agent itself. UI-TARS and OpenCUA develop native GUI agents with screenshot-based perception, action modeling, and deliberate reasoning (Qin et al., 2025; Wang et al., 2025). Agent-S and CoAct further improve performance through stronger reasoning, memory, and Best-of-N sampling (Agashe et al., 2024; Song et al., 2025), while OSExpert shows that exploration-based learning can help agents acquire environment-specific skills (Liu et al., 2026). However, existing work largely treats the UI as fixed and places the burden of adaptation on the agent. In contrast, we treat interface design as an optimization target, shifting part of the burden to the environment to better support reliable human-agent collaboration.

Method. Reliable automation with CUAs depends not only on agent capability, but also on whether interfaces expose states, actions, and transitions in ways that agents can consistently perceive, interpret, and execute. We therefore treat interface design not as a passive backdrop, but as an active factor shaping agent generalization. Our goal is not to replace classical human-centered usability principles, but to revisit and extend them for a setting in which interfaces are shared by both humans and agents, supporting effective collaboration while preserving human supervision. More broadly, we envision future digital workflows as human-centered and AI-assisted. To support this vision, we aim to design interfaces that agents can generalize to reliably while remaining understandable to humans and providing clear workflows for oversight. In this work, we proceed in three steps: first, we revisit Nielsen’s ten usability heuristics through the lens of computer-use agents (Section § 3.1); second, we offer general design suggestions for considering automation goals and agent compatibility early in the design process (Section § 3.2); third, we operationalize these ideas into concrete CUA-specific heuristics that address agent perception, grounding, and control, enabling more robust generalization across environments (Section § 3.3).

Nielsen’s ten usability heuristics provide a compact and influential framework for humancentered interface design, covering principles such as clarity, consistency, error prevention, and user control, with the full list shown in Figure 1 (Left). Although these heuristics were not originally developed for computer-use agents, we find that several transfer naturally to agent-compatible interface design, since interface properties that help humans understand interface states and follow effective workflows often also align with agents’ priors and help them as well. In particular, we identify a subset of heuristics that are directly beneficial to CUAs. First, consistency and standards help establish shared structures across interfaces, including common layouts, controls, and descriptions, which support knowledge reuse and improve generalization across applications. Second, flexibility and efficiency of use also matter for agents, as efficient workflows reduce unnecessary interaction steps; this is especially important given that current CUAs still lag behind human experts in execution efficiency (Abhyankar et al., 2025; Liu et al., 2026). Third, recognition rather than recall is highly relevant because agents rely on limited context windows to maintain interaction history, making explicit interface cues much easier to use than reconstructing prior states or procedures from memory. Finally, as with humans, error prevention and mechanisms that support error diagnosis and recovery help guide CUAs toward correct workflows and reduce avoidable failures. Taken together, these heuristics can be viewed as directly transferable principles that are broadly beneficial for CUA performance.

While several of Nielsen’s heuristics align naturally with the needs of computer-use agents, others do not transfer as directly. CUAs interact with interfaces under constraints that differ fundamentally from those of human users, including discrete screenshot-based perception, bounded contextual memory, imperfect action grounding, and limited access to implicit shortcuts or recovery strategies. Under these constraints, interface properties that remain intuitive for humans may still introduce ambiguity or execution difficulty for agents. We therefore find that blindly applying some classical heuristics can create unintended barriers for CUA performance. As summarized in Table 1, these heuristics are not unhelpful in themselves; rather, their original human-centered formulations do not fully account for the perception, grounding, and control requirements of agent interaction.

Conclusion. In this paper, we argue that improving computer-use agents should not be treated solely as a model-side problem, but also as a problem of agent-compatible interface design. From this perspective, we revisit Nielsen’s usability heuristics through the lens of CUAs, identify where classical human-centered design transfers and where adaptation is needed, and propose a human-centered, agent-compatible framework with two general design suggestions and four concrete heuristics. To study this question, we introduce UI-Verse, a controlled benchmark for analyzing how interface design affects agent behavior, and evaluate multiple CUAs on interfaces with and without our augmented heuristics. Results show that agent-compatible interfaces can consistently improve task success and often efficiency, while human studies suggest these gains can be achieved without clear usability regressions. Overall, we view this work as an initial step toward co-designed digital environments that are usable for humans and dependable for general CUAs, enabling more robust automation and more effective human–AI collaboration in both everyday and professional workflows.