Comparing Human Oversight Strategies for Computer-Use Agents

LLM-powered computer-use agents (CUAs) shift users from direct manipulation to supervising an unfolding action sequence, yet existing oversight research mostly evaluates static decisions rather than real-time oversight across an agent trajectory. We compared four oversight strategies with 48 participants across 192 live web sessions. Quantitative results show that oversight strategy affected exposure to problematic actions, but not users’ ability to intervene. Behavioral analysis shows why: control often returned too early or too late, creating a timing mismatch within the action sequence. Even when control arrived at the right moment, users judged whether the agent acted correctly, not whether the action was safe. Front-loading oversight into a plan did not solve this either: plans constrained unplanned actions but left unlisted safeguards unaddressed, while upfront approval inhibited runtime scrutiny. These findings show that oversight depends less on maximizing control than on aligning authority, timing, and attention with decision-critical moments.

Introduction. Computer-use agents (CUAs) are LLM-powered systems that perceive graphical user interfaces (GUIs), interpret interface content, and execute multi-step workflows on a user’s behalf, such as filling out forms or completing account-management tasks [6, 37]. Their emergence is shifting users from direct manipulation to supervisory coordination, creating a new challenge: users must oversee agent behavior in environments that are visually complex, semantically ambiguous, and sometimes adversarial [6, 12, 48]. Because CUAs are generative and non-deterministic, unsafe or misaligned actions can arise in ways that are difficult to notice, interpret, and correct in time [22, 48]. Recent systems have introduced approval dialogs, step-by-step controls, monitoring panels, and explanation views to support oversight during agent execution [7, 36, 38, 55]. However, existing oversight research largely treats oversight as a review problem [18, 47]: the system presents a discrete decision, the user inspects it, and then approves or corrects it. CUA oversight is instead a coordination problem over a continuously unfolding action sequence, where the timing of returned authority can matter as much as the information shown to the user. Although recent work has begun to compare specific mechanisms empirically [24], we still lack a systematic comparison of recurring oversight strategies during live, sequential execution. This matters because CUA oversight adds a stage that decision-support research does not have: the oversight strategy determines not only how well a user judges a decision, but whether a problematic action becomes available for human judgment. Prior work on reliance and explanation predicts that more control will not improve judgment, it does not predict that the strategy will change what there is to judge [5, 23, 41]. Across these systems, two coordination choices recur: who holds default decision authority during execution, and whether oversight occurs at the plan level, the action level, or both. We call these delegation structure and engagement level, drawing on mixedinitiative interaction [26], adjustable autonomy [35], and supervisory control [41]. Using these dimensions, we identify four representative strategies (Fig. 1): Action Confirmation, representing Claude Computer Use [11], defaults to requiring approval before each action; Risk-Gated oversight, representing Operator [38], allows autonomous execution with selective escalation; Supervisory Co-Execution, representing Magentic-UI [36], emphasizes plan-level engagement with runtime supervision; and Structurally Enriched, our proposed synthesis, combines plan- and action-level engagement with localized risk-aware signaling. These strategies let us compare how oversight outcomes vary with where and how human control is exercised within the action sequence, rather than treating control as a single scalar. We evaluate these strategies through three research questions on subjective experience (RQ1), problematic-action occurrence and intervention (RQ2), and delegation, inspection, and intervention behavior (RQ3). We conducted a within-subjects study with 48 participants completing live browser tasks on high-fidelity, researcherbuilt websites, each containing one embedded problematic action drawn from privacy leakage, prompt injection, or dark-pattern scenarios. We complemented the experiment with video coding of all 192 task sessions and follow-up interviews. Results show that the clearest behavioral differences lay in whether problematic actions occurred rather than in whether users successfully intervened. Across 192 sessions, plan-based strategies reduced the odds that a problematic action occurred by 76% relative to non-plan-based strategies, but users blocked only 29 of the 138 problematic actions that reached execution (21%), with no reliable difference across strategies; even under Structurally Enriched, the strategy with the lowest failure rate, 46% of sessions still ended with an uncorrected problematic action. Behavioral analysis reveals a sequence of breakdowns behind this pattern. First, control was often returned to users too early or too late relative to when it was needed. Even when control arrived at the right moment, users often evaluated whether the agent had acted correctly rather than whether the action itself was safe – a distinction we term procedural correctness versus content appropriateness (Sec. 5.2.2). Moving oversight earlier to the plan stage did not resolve this problem: plans constrained unplanned actions but left unlisted safeguards unaddressed, while upfront approval could reduce scrutiny during execution. These findings underscore the need for rethinking CUA oversight that aligns human attention and authority with decision-critical moments rather than simply maximizing opportunities for control.

Related work. Research on human oversight in automated systems has long shown that preserving human authority does not, by itself, ensure effective intervention. Classic work on automation identified a central paradox: as systems assume more routine control, human operators become less engaged and less prepared to step in when failures occur [2]. Supervisory-control research further showed that monitoring-oriented roles are vulnerable to automation bias and complacency, attentional tunneling, and out-of-the-loop performance degradation [17, 39–41]. Together, this literature suggests that effective oversight depends not only on retaining formal control, but on remaining able to recognize and respond to emerging problems [32]. Recent work extends these concerns to AI-mediated decision making. Effective oversight requires access to relevant system behavior, meaningful power to intervene, and sufficient cognitive capacity to exercise that power [47]. Faas et al. [18], for example, identify requirements such as understanding responsibilities, gaining insight into AI behavior, and contributing meaningfully to decisions. Human-AI decision-support research similarly shows that more transparency, explanation, or nominal control does not reliably improve judgment: explanations can increase acceptance without improving error discrimination [3], cognitive-forcing interventions can reduce overreliance at the cost of added effort [5], and contestability or interpretability mechanisms can alter fairness perceptions, trust, or reliance without reliably improving decision evaluation [8, 20, 28, 53]. Work on appropriate reliance further emphasizes users’ ability to distinguish when AI advice should be accepted or rejected once it is presented [45]. CUA oversight adds an earlier stage: the oversight strategy can determine whether a problematic action becomes available for human judgment. Intervention therefore requires more than preserving authority. Users must encounter a meaningful decision point, recognize why it warrants attention, and decide whether to act. Naturalistic decision-making research suggests that intervention often relies on pattern recognition under uncertainty [30], while sensemaking research shows that noticing an anomaly is insufficient unless users can interpret why it matters [29]. Recent empirical work on agentic systems likewise shows that oversight spans multiple stages. Dhanorkar et al. [13] identify a priori control, co-planning, realtime monitoring, and post hoc review as distinct forms of oversight work in developers’ use of software agents.

Method. We conceptualize oversight strategy as a structural property of human-agent coordination along two dimensions: delegation structure, which captures where default decision authority resides, and engagement level, which captures where human oversight enters the workflow. Fig. 1 in Sec. 1 summarizes this conceptual map; we explain it in detail below and use it to organize the four strategies compared in this study.

Delegation structure distinguishes agent-led from human-controlled execution. In an agent-led strategy, the agent selects and executes actions by default and escalates only under specified conditions. In a human-controlled strategy, execution requires user authorization. Engagement level captures whether human oversight occurs at the plan level, the action level, or both. We treat plan- and actionlevel engagement as independent because they support different forms of intervention. Plan-level engagement lets users inspect or revise the entire task structure before execution begins, when the full plan is visible at once and there is little time pressure to decide; action-level engagement instead requires users to decide on individual actions as they arise during execution, typically under time pressure and with visibility limited to the current step, so earlier context must be recalled rather than re-inspected. A strategy may therefore support either or both forms of engagement. Fig. 1 represents delegation structure as an axis and engagement levels as separate tags, allowing a hybrid strategy such as Structurally Enriched to engage at both levels without being forced into a single intermediate position. Using this map, we instantiate four strategies for comparison: three abstracted from prior systems and one structurally enriched strategy that combines multiple forms of engagement. Each condition operationalizes a coordination strategy as a bundle of interface and control mechanisms; our goal is to compare strategy-level structures holistically rather than estimate the causal effect of individual UI components. Delegation structure and engagement level are not crossed factorially across our four conditions: three strategies are agent-led and only one (Action Confirmation) is human-controlled, and Structurally Enriched combines multiple engagement mechanisms at once. Fig. 1 should therefore be read as an organizing map that motivated the selection of four representative, bundled configurations, not as a factorial design whose two dimensions are independently estimated; we return to this scoping in Sec. 6.5.

3.2.1 Risk-Gated. Risk-Gated oversight represents an agent-led delegation structure with selective escalation. The agent executes autonomously and involves the user only when it identifies an action as potentially consequential or risky. This pattern is inspired by OpenAI’s Operator [38], which requests user confirmation or returns control at consequential moments while allowing routine execution to continue autonomously. Human engagement is therefore sparse and concentrated at system-selected action-level decision points rather than distributed across the full execution stream. Our implementation retains this selective-escalation logic but uses a modal approval dialog rather than a full browser handoff. The dialog also provides an “Approve Similar” option that pre-authorizes future actions of the same coarse type, e.g., any subsequent textentry action regardless of field or page, rather than only the specific field or page the user just approved. This strategy minimizes interruption, but its effectiveness depends on whether system-selected escalation points coincide with moments users themselves would consider worthy of review.

3.2.2 Action Confirmation. Action Confirmation uses a humancontrolled default: each proposed action requires explicit user approval before execution. This pattern is inspired by per-action confirmation patterns in Anthropic’s Claude Computer Use [11]. Human oversight is therefore organized at the action level and enforced at every action boundary by default. Our implementation retains this halt-and-approve logic and presents a brief summary of the proposed action before confirmation.

Discussion. Human oversight for CUAs should be adaptive rather than fixed. Existing systems instantiate different oversight strategies—from confirmation around sensitive actions to co-planning, co-execution, and per-action approval [11, 19, 36, 38, 55]—but typically commit to a particular coordination pattern or predefined class of escalation. Our results suggest that no single strategy is uniformly appropriate: perceived fit varied across task contexts, while more frequent opportunities for approval did not reliably improve runtime intervention. The relevant design question is therefore not which oversight strategy is best in general, but which form of oversight is appropriate for the current context. Oversight may need to adapt its timing, granularity, and information support as the task, environmental state, prior delegation, and reversibility of consequences change. This extends a long-standing insight from human-automation research. Classic work has shown that preserving formal authority is insufficient when automation reduces users’ engagement and preparedness to intervene [2, 17, 40], while mixed-initiative interaction, adjustable autonomy, and levels-of-automation research argue that control should shift across actors and stages of a task [26, 35, 41]. CUA interaction introduces a further challenge: as agent execution becomes increasingly long-horizon, returning control is not enough. By the time a user is asked to intervene, the agent may already have taken multiple actions, altered the environment, relied on prior approvals, and moved the task into a state whose significance cannot be understood from the current action alone. The goal of oversight should therefore shift from keeping humans continuously “in the loop” to scaffolding effective human re-entry. When users re-enter an evolving agent trajectory, they need more than a description of the next action. They need enough context to understand how the interaction reached the current state, what consequential change is now at stake, and what boundary of future behavior their decision will authorize. This makes re-entry a sensemaking problem as much as a control problem. Situationawareness theory distinguishes perceiving information from comprehending its meaning and projecting its consequences [16], while sensemaking research similarly shows that noticing a cue is insufficient without understanding why it matters [29]. For CUAs, this interpretive burden is intensified because the relevant evidence is distributed across a trajectory of agent actions, environmental states, and prior delegation decisions. We call the support needed at these moments re-entry scaffolding. Rather than presenting an isolated action for approval, such scaffolding should reconstruct the decision context around it: the relevant antecedents, the consequential state transition, and the decision boundary—what exactly is being authorized, what remains reversible, and how far that authorization extends into subsequent execution. The objective is not more frequent human interruption, but better-timed and better-supported human judgment.

6.2 Aligning Human Judgment with Consequential State Changes Re-entry scaffolding is useful only if it brings human judgment to bear on the decision that actually matters. We call this requirement judgment alignment: aligning human attention and authority with the timing, interpretation, and scope of a consequential state change. This framing extends prior work on adjustable autonomy and autonomy alignment, which asks when control should shift between human and system [35, 57]. For long-horizon CUAs, however, returning control is only the first step: the returned decision must also correspond to the state transition the user needs to evaluate.

Conclusion.

Limitations. 6.5 Limitations and Future Work Our findings should be interpreted within several limitations that also motivate directions for future work. First, task context was partially confounded with risk type. Higherconsequence tasks primarily involved privacy leakage, whereas lower-consequence tasks involved prompt injection and dark patterns. The observed context effects may therefore reflect differences in consequence, risk type, workflow structure, or how risks become legible during interaction. As reported in Appendix I.3, variation across scenarios within each context group was comparable to the difference between the two group means. We therefore treat the higher-versus-lower contrast as descriptive of the task contexts studied rather than as an isolated causal effect of consequence level. Future studies could orthogonally vary consequence level, risk type, and workflow structure across matched scenarios to identify which contextual factors should drive adaptive oversight. Second, intervention is necessarily conditional on a problematic action occurring. This creates a selection effect: episodes available for intervention may disproportionately contain risks that were harder to prevent upstream or that surfaced later in execution. Intervention rates should therefore not be interpreted as unconditional measures of users’ overall oversight ability.