Overseeing Agents Without Constant Oversight: Challenges and Opportunities

To enable human oversight, agentic AI systems often provide a trace of reasoning and action steps. Designing traces to have an informative, but not overwhelming, level of detail remains a critical challenge. In three user studies on a Computer User Agent, we investigate the utility of basic action traces for verification, explore three alternatives via design probes, and test a novel interface’s impact on error finding in question-answering tasks. As expected, we find that current practices are cumbersome, limiting their efficacy. Conversely, our proposed design reduced the time participants spent finding errors. However, although participants reported higher levels of confidence in their decisions, their final accuracy was not meaningfully improved. To this end, our study surfaces challenges for human verification of agentic systems, including managing built-in assumptions, users’ subjective and changing correctness criteria, and the shortcomings, yet importance, of communicating the agent’s process.

Introduction. Automated agents, discussed as an interaction technique for decades [49, 53, 73, 74], have gained traction for use in complex tasks following recent advances in large language models. Coding agents like Claude [6] and Github Copilot [32] write code and use code execution engines, Deep Research tools produce detailed reports [33, 61], and Computer Use Agents (CUA) like Operator [63] and Fara- 7B [7] interact with the web. This rapidly increasing suite of publicly available agents aims to offload tedious and difficult tasks from the user to the agent [3]. However, these systems remain error-prone, pose privacy and safety concerns, and their success is often context dependent [18, 20, 66]. To mitigate these risks and retain human agency, an expanding body of work incorporates human-in-the-loop (HIL) features into agentic systems [92]. For example, the HIL CUA Magentic-UI enables users to edit the plan of action before it occurs, take over the browser at any point, and approve of consequential actions. With such control, a user performing a task like buying tickets to local museums can change the plan to prefer the museums’ official website over third parties, perform navigation actions if the agent struggles, and give explicit approval before payment. Before a user can take control to correct an error or provide clarification, they must find and define the problem. In the above scenario, if the system selects tickets for the wrong day, or if the museum is actually closed on the user-specified day, steering the output to avoid these complications requires first recognizing that they occur. To make the agent’s actions and errors visible to the user, current systems output information about the agent’s progress. Such information includes an overview of planned steps, text descriptions of actions taken and their motivations, summarizations of sets of these actions, citations to references, and screenshots of the browser if applicable [31, 33, 58, 60]. We use the term trace to refer to this provided information. To be effective, a trace should enable users to find errors accurately and without expending needless effort. Unfortunately, extensive prior work finds that explaining AI behavior to users often does not improve comprehension, decision making, or appropriate reliance, thus necessitating careful design [10, 16, 39, 79, 80]. Initial work suggests that such comprehension remains challenging for agentic workflows. One study found that, of all interaction types, participants spent the most time on output inspection [29], and communication errors occurred in 65% of cases with a real user in the Collaborative Gym benchmark [70]. With cumbersome communication, users spend less effort verifying or take longer to verify than they would to do the task themselves, reducing the utility of the overall system [81]. We investigate and improve upon agentic trace presentation for the CUA Magentic-UI [58] in a series of user studies (Figure 1). We first ran a formative study (n=12) to identify how users verify with existing supports. We found that the current trace design is verbose, cumbersome, and timeconsuming, with participants missing small but impactful errors. Aiming to improve user experience and error-finding capabilities, a subsequent study (n=12) tested the efficacy of three design probes that summarize the trace content: Flowchart, Citation, and Specification summary. Participants were presented with an already-concluded task and were asked to determine if the system’s answer to a question was correct. Flowchart, the most process-oriented probe of the three designs, had the lowest error-finding rate and was the least preferred by participants. Specification had the highest error-finding rate and was most preferred, while Citation had middling performance. In a final study, we incorporated our prior findings into a single novel interface. This study determined that participants (n=12) found errors faster (Hedges’ g: -0.65) with our interface. However, we found no meaningful effect on improving overall accuracy (Hedges’ g: 0.18). More worrisome, when errors were missed, participants reported higher confidence in the agent’s correctness (Hedges’ g: 0.85). Our iterative design process characterized crucial but under-served challenges for human validation of multi-step reasoning models and agentic workflows. For instance, when participants mistakenly thought an incorrect answer was correct, they often cited a “reasonable” process taken by the agent as their justification. Presentations that highlight the overall process rather than the execution of each step exacerbated this overreliance. However, participants also felt lost when the presentation was divorced from the process’s context. Thus, there is a tension between communicating the overall process while grounding in execution details.

Related work. Human-in-the-loop agentic systems intend to provide improved performance as well as user context and agency while still reducing effort (2.1). Human-AI interaction studies show the necessity of carefully designed interventions for verification and oversight, but existing automated methods for measuring CUA, namely benchmarks, rarely explicitly study HIL performance (2.2). We investigate how well users interacting with a HIL CUA system are able to find errors, and we illuminate opportunities and challenges for improved oversight through iterative design.

Improved model capabilities have powered the rapid development and deployment of AI Agents [2, 5, 63]. The AI Agents Directory now holds 2000+ agents and is quickly expanding [3]. Despite this rapid growth, current agents have substantial deficiencies in performance and can be costly [38, 42, 70, 89]. A human-in-the-loop approach, in which a person can review and steer the agent’s actions throughout execution, intends to improve performance while reducing effort for the user overall [92]. For instance, OpenAI’s Operator [63] allows users to take control of the browser and make manual interventions if necessary. Beyond error fixing, human oversight also enables incorporation of user context, preferences, and expertise [38, 39, 70, 76] as well as adaptability to changing requirements from criteria drift [69] and unexpected contingencies [83]. Another subset of work argues for semi-autonomous systems to retain human agency [12, 18, 22, 30, 47, 57, 93]. The HCI community has discussed and designed semi-autonomous agents for programming and interacting with the web for decades [49, 53, 73, 74]. As the necessary technology was not always available, these studies often employed Wizard-of-Oz techniques [54]. More recent studies address human interaction with implemented HIL agentic systems, including Cocoa [31], CowPilot [38], and Magentic-UI [58], finding that HIL systems enable better performance than entirely autonomous approaches. For instance, Shao et al. found that Human-Agent collaboration improved performance for both simulated users and real users, although communication challenges increased the number of steps needed to complete the task in the simulated condition [70]. Another branch of emerging work visualizes task decomposition from generative AI [43, 66]. Outside of end-user steering, prior work supports human and automated oversight of workflow execution. Such research for multi-agent workflows helps developers debug errors to improve workflow design [24, 28, 64, 85, 86] and incorporates LLM-based self-verification into workflows [27, 67, 69, 84], although building effectively self-verifying workflows remains challenging [17, 35].

Method. Human oversight and verification of agentic workflows is an important feature for safety, agency, and performance [18, 22]. However, initial work finds that communication errors are prevalent [70] and that users spend the most time on output reviewing [31]. This formative study aims to set a foundation of existing verification practices and pain points, particularly when multitasking. This section first reviews the CUA system, Magentic-UI [58], that we use (3.1), our study procedure (3.2), our findings (3.3), and takeaways to inform our design probes (3.4).

3.1 System Magentic-UI is an interface for HIL oversight of CUA that can interact with the browser, a code execution environment, and a file system in a contained Docker environment. It incorporates several HIL features, including: co-planning (users can review and edit the plan before it executes), co-tasking (users can take over the live browser), action approval (users must approve before the system takes an irreversable action), multi-tasking (multiple runs can co-occur), and final answer verification (a process summary with the final answer). As shown in Figure 2, the interface displays a text trace describing the task [A], the CUA’s planning steps [B], each individual action [C], and mid-level summaries [D]. After execution, a “Final Answer” summary appears here. The other panel displays an interactive live view while the CUA uses the browser [E] and screenshots taken at regular intervals [F]. Relevant action text [C] contains an icon that, when pressed, navigates to the associated screenshot [F]. Users can move between multiple ongoing and previous tasks [G] and interact through chat [H].

Fig. 2. Magentic-UI interface. Our formative study illuminates verification practices with the HIL CUA Magentic-UI [58]. Users first input a task [A], then use various features to co-plan, co-task, and adapt to errors as the system runs. To verify, they can first review the plan summary that decomposes the task into proposed plan steps [B]. Each plan step is then executed, and each action is described in the trace for transparency [C]. Collapsing these plan steps [D] shows summaries of each group of actions. While the task runs, users may watch the live view as the agent interacts with the browser [E]. This visual trace is stored in a series of screenshots as the agent progresses [F]. Clicking the icon next to a particular action [C] links to the associated screenshot [F] to improve navigation. Users can open tabs between different tasks to multitask [G] and interact with chat [H]. A “Final Answer” summary appears upon completion (not visible). signed a consent form, then viewed a demonstration of Magentic-UI features. Participants were then given the opportunity to explore features by running a task directed towards finding publications. Next, participants completed three tasks simultaneously to encourage multi-tasking behaviors. Inspired by tasks on AssistantBench [90], we created three tasks that sometimes caused errors and often triggered Magentic-UI’s HIL features. These tasks were:

Task 1: “Add a Hawaiian Pizza from TangleTown to my cart.” Task 2: “Find all the appetizers with tuna from JustOneCookbook and save them to an alphabetized csv file with the recipe name and url.” Task 3: “How much will I save by getting annual passes for my family (2 adults, 1 kid age 5, 1 kid age 2) for the Seattle Aquarium, compared to buying daily tickets, if we visit 4 times in a year?”

Participants then ran a task of their own choosing. Next, participants completed the System Usability Scale [15] and a questionnaire about their familiarity with agentic systems. Finally, participants took part in a semi-structured interview. Each study was one hour long. Participants were encouraged to think aloud throughout the study. By hand, we extracted salient quotes from participants’ think-aloud statements and semi-structured interview responses, then performed five rounds of inductive and deductive thematic analysis to organize them into themes [14].

Discussion. 6.1 Error finding requires detail, but lists of actions are overwhelmingly verbose. Supports should provide navigation for this detail to instill appropriate confidence. We find that participants miss errors and feel overwhelmed when reviewing long traces of an agent’s series of actions. Summarizing sets of steps reduces this verbosity but can hide execution errors. Thus, determining the correct level of granularity for reporting agent actions is a crucial challenge [9, 29, 41, 71, 75]. Rather than adjusting the granularity of the step detail, our approach supports user navigation of execution detail with a higher-level specification to reduce user effort while staying grounded in execution steps. However, there remain many open questions towards supporting this navigation. Our treatment interface had a high learning curve and increased confidence, especially when users were wrong. Future investigations can study verification practices with more ecological validity to everyday use of these agents. In our controlled study, participants had access to the browser, so many executed the task themselves. This ability to complete the task themselves may have evened performance between the two conditions. However, several participants desired lightweight validation because they would not extend such effort in everyday life, especially for low-stakes tasks. Future work can also investigate if this higher confidence translates to worse decision-making and study the integration of methods to reduce overreliance, such as cognitive forcing functions [16]. We included checked requirements as a part of the design to improve scanability. However, incorrectly confirmed requirements might contribute to higher confidence as they are a highly visible set of green checkmarks. Further iteration on requirement and assumption content and implementation could improve existing outcomes. Finally, the space of opportunities for trace navigation depends on the system’s ability to generate these requirement, assumption, and annotation guides.

6.2 Process provenance matters but is challenging to effectively display. Across all studies, participants relied on the overall process and cared about the quality of information sources. However, a high-level review of the process could hide details in the execution of individual steps. In the design probes study, participants often referred to a “reasonable” process when justifying why they thought an incorrect answer was correct. Despite this risk of overreliance, many participants found interpreting the results without the process’s global context to be challenging. The Specification method had the highest accuracy, but it took the longest, required the most checks of the overall trace, and some participants lamented its disconnect from the process. To navigate this tension, our final design’s main feature was the outcome-oriented Specification approach linked directly into the trace. Participants justified overreliance with a reasonable process less often in the controlled study than in the design probes study. Future work could investigate if the challenge of being divorced from the process might positively encourage more cognitive engagement and run additional control studies on content inclusion for different presentation methods, such as integrating assumptions into a flowchart design.

6.3 Trace displays can extend to show what the agent did not do. In the open world of browser use, there can be many approaches to answering a given task, especially when the task is underspecified. Participants were aware that the CUA was choosing one path among many and wanted this broader context to be more explicitly surfaced.

Conclusion. Through a formative study, we found that people miss small but impactful errors in the verbose existing CUA system and have a subjective and changing sense of correctness. A follow-up study of three design probes revealed that people could overrely if the overall process is reasonable and that they wanted a high-level overview with drill-down capabilities. From these findings, we create a novel interface and test its impact on participants’ correctness judgments in a counterbalanced study. With our interface, users found errors faster, but had higher confidence without meaningfully improving accuracy. Our studies illuminate challenges and opportunities for future development supporting human oversight of multi-agent systems.

Limitations. Our study has several limitations beyond those discussed in the previous section. Our participants were all employees of a large technology firm who had previously used AI agents. These participants are not representative of the broader set of users with access to publicly released agents. Additionally, we only test one CUA system, Magentic-UI. Other reasoning models and agents outside of CUA similarly use long traces as explanation, so extending a similar design process to those will improve understanding of these findings’ generalizability. Our study setup also reviewed just one part of verification, post-hoc verification with an assumed correct or incorrect answer. Finally, our metric of confidence was a self-defined scale from 1 to 7. Further exploring this strong effect with a more nuanced metric of confidence would better define this issue. We ran a Wizard-of-Oz study to enable rapid iteration on the system design. However, full implementation of this interface would require automated surfacing of the abstractions. After predicting requirements as a part of the planning stage, automated judges could track if requirements are completed and what assumptions occur, similar to how Magentic-UI’s action guard acts as an external evaluator.