Do developers actually review agent work exhaustively?

Dhanorkar, Passi, and Vorvoreanu's interview study of 17 experienced developers finds that oversight "functions not through exhaustive awareness, but through practical heuristics." Existing oversight research "largely aspires for 'effective' oversight by facilitating continuous, 'meaningful' human control over AI and agents," but the authors report that "such aspirations often prove unscalable and are decoupled from the practical realities of users running software agents." During post hoc review, participants instead opted for "good-enough supervision driven by 'bounded rationality' rather than exhaustive awareness of agentic outputs and workings." The paper names three concrete heuristics: "Plans become proxies for agentic working, test results stand in for code correctness, and skimming becomes the main information processing mechanism."

The mechanism the authors give is bounded rationality under real conditions of complexity and time pressure: faced with agent outputs too voluminous or opaque to review in full, "developers adopt emergent heuristics that prioritize functional efficiency over idealized perfection." Rather than independently verifying what an agent did, a developer treats a passing test suite as a stand-in for correctness and a previously agreed-upon plan as a stand-in for whether the agent's working matches intent — shortcuts that substitute a cheaper signal for the thing oversight is nominally supposed to establish.

This qualifies the normative literature on oversight that the vault otherwise holds from the design or governance side. Does granting agents more autonomy undermine human oversight? argues that agent design erodes the situational awareness and judgment oversight requires; this study suggests developers are not simply losing that capacity passively but are actively substituting cheaper heuristics for it, which could look like erosion from outside while developers experience it as functional adaptation. It also puts pressure on Can execution traces ground honest explanations of agent behavior?, which proposes a structured, trace-grounded report to support thorough post hoc review: this study's developers do not reach for execution-trace detail by default, instead skimming and trusting test results, so a tool that offers richer traceability may go underused unless it fits inside the heuristic rather than replacing it. The companion note How do developers actually oversee AI coding agents? describes when this heuristic-driven review happens in the oversight timeline; this note describes how it actually gets done.

The excerpt does not say how often these heuristics catch real mistakes versus let them through, nor does it quantify skimming, nor does it generalize beyond the study's power users on relatively small, bounded tasks at well-resourced firms. The authors flag this directly: participants "delegated relatively small tasks to agents, rendering, for example, real-time monitoring optional and post hoc review easier," and call for research on heuristics under "complex, longer-running tasks." What follows at the strength the excerpt supports is a design implication rather than a safety verdict: tools and governance frameworks that assume thorough, exhaustive review should account for the fact that experienced users default to cheaper proxies, and ask whether those proxies are reliable enough for the task at hand rather than assuming oversight happens as specified.