The Oversight Fallacy: Why AI Agents Require More than Humans-in-the-Loop

Source: Samir Passi, Ranjit Singh (Data & Society); Research report (fieldwork in scientific labs using agents) · 2026

In 2026, autonomous AI agents have begun to move from speculative promise into daily deployment. Unlike chatbots, which primarily respond to user queries, AI agents are designed to take a user’s high-level goal, translate it into a plan, and act across digital environments. These agents can navigate websites, run code, analyze files, edit documents, call tools, and coordinate multi-step tasks. For AI companies and startups, agents represent the next phase of AI development.

The relative autonomy of AI agents challenges the traditional means of human oversight. Oversight is the work of supervising agents so their failures can be recognized and addressed. When a user gives an agent goals, they also give it room to decide how those goals should be pursued. An agent makes and carries out those decisions through tools and connected systems. We call the space between what the agent is asked to do and what it does the goal-plan-execution gap. When the user’s goal and the agent’s execution drift apart, failures can become consequential, from altered data to deleted files or misallocated resources. Because agents move fast and can act across many systems at once, a small mistake can quickly create ripples.

In 2025, OpenAI announced that ChatGPT would begin offering “agent mode,” claiming that it could handle “complex tasks from start to finish.”2 The marketing copy of this announcement emphasized the two major capabilities that distinguish AI agents from other systems: agents can generate multi-step plans from simple, plain-language goals, and agents are connected to other software systems so they can actually act on those plans. In public applications like ChatGPT, this largely means interacting with the web: navigating websites through a browser, using connected tools and services, and logging into accounts when users authorize access.

Most importantly, you’re always in control.

Here, we get a tidy encapsulation of the promise of human oversight. OpenAI suggests, as many believe, that oversight is accomplished by putting a human in the loop. Users are imagined to be always in control, there to prevent harm before it happens. But oversight of AI agents is neither easy, nor natural. For users to be able to oversee agents, builders must deliberately design actionable mechanisms that make it possible to do so. The challenge of creating these mechanisms, and the work of using them appropriately, is what this primer examines.

Human oversight will always be necessary for AI agents, because failure is a normal and ordinary part of agentic systems. This is because of the challenges in translating goals into plans and then putting those plans into action. We call this the goal-planexecution gap.6 Each transformation — from goal to plan, and plan to execution — creates its own conditions for failures.7 Agents can misinterpret user intent, make incorrect assumptions, lose context, mishandle edge cases, or fail when using tools and services.8 Even when agents appear to succeed, success can be partial; agents may lose the trail of key information and decisions, miss relationships between sub-tasks, or cut corners to ensure task success.9 The more seamless the system, the harder it is to notice failures. Better agents will make these failures less frequent, but not fully eliminate them. In fact, highly reliable agents that work correctly most of the time can be harder to oversee than unreliable agents because the former makes it easier to get complacent and harder to notice when things can and do go wrong.

For example, software company Sakana AI introduced “The AI Scientist” in 2024 to automate scientific research. The company claimed that the AI Scientist could do every step of a scientific workflow: generate ideas, run experiments, analyze data, and write up results.10 However, while testing the system, the team ran into several failures. One involved timeout limits: users could limit the total time the AI Scientist could take to run a particular experiment; if the process took too long, the system should stop and report. Yet during testing, when “experiments exceeded [the] imposed time limits, [the system wrongly] attempted to edit the code to extend the time limit arbitrarily instead of [taking the right approach of] trying to shorten the runtime.”11 The system arrived at a final result by bypassing, rather than complying with, the experimenter’s constraint.

We propose a four-part framework for specifying the conditions that make effective AI agent oversight possible.

The more seamless the system, the harder it is to notice failures.

Agent builders and policymakers treat the user as a safety mechanism in agentic systems, expecting users to intervene in the outputs and working of agents to identify, fix, and prevent mistakes and failures. Users are given pause buttons, approval checkpoints, undo options, warnings, or reminders to review outputs. But treating intervention as the main site of oversight weakens the conditions that make intervention meaningful in the first place. To intervene effectively, users must first be able to know, observe, and control agents effectively.13 A pause button does not help unless the user can tell what the agent has done and when it must be paused.

This creates a tension at the heart of agent design. Agents are often marketed and built around seamlessness: less friction and more autonomous completion of work. Oversight, however, requires a different design priority. It requires deliberate points of friction where users can inspect, contest, redirect, or halt agentic action before small divergences become consequential failures. The design problem is deciding exactly when and where friction belongs in agentic systems, and how to implement it strategically to keep delegation accountable without overwhelming users.

This primer explains how and why knowing, observing, and controlling modern agentic systems is currently exceptionally challenging for users across three key oversight practices: post-hoc auditing, real-time monitoring, and steering. Although users are often positioned as the last line of defense against agentic failure, we argue that dominant approaches to the design and regulation of oversight frequently leave them without the practical capacity to supervise agents effectively. Addressing this problem requires moving away from the singular human-in-the-loop solution. Oversight cannot rest on the user alone; it has to be organized as a shared responsibility between builders and deployers. Ultimately, human oversight of AI is a problem of communication between humans as much as it is a problem of control over AI systems.

One of our fieldwork sites is a computational biology lab where biologists and computer scientists evaluate how AI agents perform ordinary laboratory tasks. The team benchmarks agent performance by building repeatable tasks and evaluation criteria that allow them to compare how different agent systems, including Claude Code and Codex, perform across runs, versions, prompts, and task complexity. Their ultimate research question is: when given a well-scoped research task that resembles ordinary lab work, can an agent complete that task in a way that scientists would recognize as methodologically acceptable?

KNOWLEDGE concerns what users must adequately understand about system outputs, capabilities, working, limits, controls, failure modes, and (un)intended use cases. Users need a realistic mental model33 of agentic systems to, for instance, anticipate mistakes, determine necessary reviews, and identify critical workflow aspects requiring attention.

OBSERVATION requires users to sufficiently see and interpret system parts. For agentic systems, users must scrutinize not only final outputs but also intermediate steps, assumptions, retrieved information, tool use, and the evolving workflow through which outputs are produced.

CONTROL requires users to meaningfully direct system working. This includes limiting an agent’s tool use and data access, requiring user confirmation for specific agent actions, and forbidding unacceptable or undesirable courses of action.

INTERVENTION requires users to step in and identify and fix issues during and after agent execution in a timely manner. Intervention involves pausing, redirecting, revising, rolling back, or shutting down agents before errors compound.

Intervention depends heavily on knowledge, observation, and control. Users must know enough about an agent, observe enough of its working, and have enough control over its permissions and trajectory before they can intervene effectively.34 Currently, however, among the agent systems we have observed, builders do not prioritize these components equally. The most common approach is to emphasize oversight in the form of intervention and at times control, while downplaying knowledge and observation. This imbalance is partly practical and partly institutional. It is often easier to add a confirmation prompt than to design a usable account of what an agent has done. It is also easier to let users configure autonomy than to explain how autonomy changes the kinds of failures they may need to notice. Building useful knowledge and observation into agentic systems requires iterative, human-centered design, yet current market incentives often reward seamless completion over inspectable work.

Post-hoc auditing is the process of retrospective observation and intervention, reviewing and correcting the output and working of an agent after execution.35 Examples of this work include verifying output accuracy, “sensemaking”36 of agent actions, and re-running agents to fix mistakes. Across research, policy, and industry, post-hoc auditing is the most frequently discussed oversight practice.37 Yet this form of intervention remains challenging to execute because it essentially depends on observation and knowledge support that are often lacking. The details of agent actions, interactions, communications, tool use, and workflows are dense and lengthy, making comprehension and review difficult. Even with everything logged, the practical issue remains: can users easily and meaningfully interpret the information provided without turning post-hoc auditing into its own cumbersome project?

Can users easily and meaningfully interpret the information provided without turning post-hoc auditing into its own cumbersome project?

At the computational biology lab we observed, one benchmark task asked agents to work with video-based fruit fly behavior data. The underlying experiment exposed flies to a small LED light and recorded their movement while the light turned on and off. The agent’s assigned goal was to analyze the resulting data: compare fly movement during LED-ON frames with fly movement during LED-OFF frames.

The problem appeared only when the team reviewed how the agent had moved from goal to plan to execution. As Ryo,39 the lead researcher on the project, recounted later, the agent had skipped the per-fly step. Instead of computing ON and OFF averages for each individual fly and then averaging those values across the group, it latched onto “one fly ... as a proxy for the entire group of flies,” treating a single heavily labeled trajectory as representative and effectively duplicating that individual’s data to stand in for everyone.40 By chance, that one fly’s speeds were close to the true group mean, so the final ON/OFF/ difference fell within tolerance, and it was marked as correct. It was “numerically right but methodologically wrong.”41 The assigned goal was appropriate, and the final output acceptable, but the agent’s execution skipped the step of preserving variation across individual flies that made the task scientifically meaningful.