Does oversight strategy prevent risky agent actions or just catch them?

A within-subjects study of 48 participants across 192 live web sessions compared four strategies for overseeing computer-use agents (CUAs) — LLM-powered systems that read a graphical interface and carry out multi-step tasks like filling forms — and found that "oversight strategy affected exposure to problematic actions, but not users' ability to intervene." Plan-based strategies, which let a person inspect or revise the agent's full task plan before execution starts, "reduced the odds that a problematic action occurred by 76% relative to non-plan-based strategies." But once a problematic action did reach execution, users blocked only "29 of the 138 problematic actions that reached execution (21%), with no reliable difference across strategies." Even under Structurally Enriched, the authors' own synthesis strategy and the one with the lowest failure rate, "46% of sessions still ended with an uncorrected problematic action."

The authors trace this gap to two independent design dimensions: delegation structure (whether the agent or the person holds default decision authority) and engagement level (whether oversight happens at the plan level, the action level, or both). The four tested strategies — Action Confirmation (modeled on Claude Computer Use's per-action approval), Risk-Gated (modeled on OpenAI's Operator, which escalates only at system-selected moments), Supervisory Co-Execution (modeled on Magentic-UI's plan-level engagement), and Structurally Enriched (the paper's own combination of plan- and action-level engagement with localized risk signaling) — differ in how much control they return, but behavioral coding of all 192 sessions shows "control often returned too early or too late, creating a timing mismatch within the action sequence." Front-loading review into the plan stage "constrained unplanned actions but left unlisted safeguards unaddressed," and upfront approval "could reduce scrutiny during execution."

This complicates the general claim in Does targeted human oversight beat both full autonomy and exhaustive review? that routing control to the right moment, rather than maximizing control, is where the leverage lies. That paper's confidence-routed CoPilot mode hit an 87.5% accept rate; here, routing attention earlier to a plan cut how often risk arose but left intervention stuck at roughly one in five once a risk reached execution, regardless of strategy — timing authority correctly is necessary but, on this evidence, not sufficient for users to act on what they are given. It extends the general argument in Does granting agents more autonomy undermine human oversight? with a controlled comparison across specific interface strategies, and runs parallel to Can organizations lose scrutiny capacity while keeping oversight forms?: formal opportunities to intervene persisted across all four conditions, but the capacity to act on them did not track which strategy was in place.

The study does not establish why intervention stayed flat across strategies — a distinct finding about what users actually judge, covered separately — nor does it generalize beyond browser-based tasks built around three risk types (privacy leakage, prompt injection, dark patterns) in a researcher-constructed environment; the authors note task context was "partially confounded with risk type" and that intervention rates are conditional on a problematic action having occurred at all, so they "should not be interpreted as unconditional measures of users' overall oversight ability." The implication the data support is narrower than "more oversight helps": added control reduced how often agents created risk, but did not, in this sample, change whether users caught it when it slipped through.