Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents

Autonomous software agents hold promise to increase developer productivity but make mistakes and exhibit novel failure modes, making human oversight central to successful human-agent collaboration. Existing research on agent oversight is largely conceptual; normative frameworks exist, but how users actually oversee agents is less known. In this paper, we bridge this gap by providing early empirical anchors for the theoretical discourse on agent oversight. Drawing on interviews with 17 experienced developers, we conduct an exploratory inquiry examining what forms of emergent oversight work developers perform, when, and how. We also document the oversight challenges developers face and the strategies they have started using to address them. We found at least four forms of emergent oversight work: a priori control, co-planning, real-time monitoring, and post hoc review. We show that oversight work is not only reactive and retrospective, as portrayed in existing research, but also preventative and proactive. We describe situated oversight challenges (e.g., difficulty reviewing agent-generated code) and outline heuristics developers adopt to address such challenges (e.g., using test results as guarantees for code correctness). We conclude with high-level takeaways, future research directions, implications for the human-centered design of software agents and for software engineering practice, and limitations of our research.

Introduction. Agentic AI systems are becoming ubiquitous, with applications across tasks such as data analysis, scientific discovery, and web navigation [14, 32, 43, 85, 93, 115]. Unlike conventional AI, agents can autonomously plan, use external tools, and take actions in real-world environments [79, 108, 116, 151]. The impact of agents is perhaps most evident in software engineering, which has seen a proliferation of tools such as Cursor, Claude Code, Cline, ChatGPT Codex, and GitHub Copilot, and in technical advancements such as Open Code Interpreter and AlphaEvolve [90, 140, 154]. Specialized code agents–systems capable of making repository-level changes [138, 149, 152]–have pushed way beyond simple code generation tools [49, 58] and made their way into developer workflows and organizational practices.

Agentic systems, however, are not without problems. They often fail for several reasons, such as misinterpreting requests, errors during interactions with external tools, inability to identify mistakes, and context loss [7, 17, 55, 82].

Autonomous coding accelerates software development but also paves the way for mistakes that silently propagate into production. Irreversible errors, ranging from localized file deletions [80] to large-scale production incidents, such as the Replit agent’s database erasures [89], pose a significant threat to system reliability. Agents also face cybersecurity risks both as vulnerable systems (e.g., susceptibility to prompt injection attacks, as in Gemini CLI [28]) and as enablers of large-scale autonomous attacks (e.g., credentials and sensitive data leakage, such as in the Claude Code incident [5]).

While automated safety methods (e.g., red teaming, agents-as-a-judge) are increasingly used to catch agentic failures [40, 125, 156], they remain limited by the scenarios and incentives they encode [102]. Agents may exhibit biases [6] and behave differently in real deployments than in evaluations [81, 87].

All of this makes human oversight of agents a priority. This is also evident in governance frameworks that increasingly mandate human oversight of AI. The EU AI Act explicitly requires that humans retain meaningful control over high-risk ∗Both authors contributed equally to this research.

Authors’ Contact Information: Shipi Dhanorkar, shipidhanorkar@gmail.com, Microsoft, Redmond, USA; Samir Passi, v-sapassi@microsoft.com, Microsoft, Redmond, USA; Mihaela Vorvoreanu, Microsoft, Redmond, USA, mihaela.vorvoreanu@microsoft.com. arXiv:2606.05391v1 [cs.SE] 3 Jun 2026 2 Dhanorkar, Passi & Vorvoreanu AI, and the European Commission’s guidance on trustworthy AI identifies oversight as a core requirement [1]. However, such frameworks assume that oversight is feasible without thoroughly examining or specifying what it looks like in practice. It often falls on users to do oversight in their context, ensure agents work properly, and fix agent mistakes and failures [1, 8, 25, 64, 114]. However, users frequently struggle to identify mistakes, even in conventional AI outputs, and exhibit overreliance [91, 95–97]. Agentic systems exacerbate these issues because they introduce new failure modes and require users to assess not only agent outputs but also agent actions [17, 82, 94, 114]. Oversight challenges are particularly acute for developers who must oversee agents across multiple software tasks and projects in production environments under time pressure.

Agent oversight is essential for effective human-agent collaboration and is increasingly seen as an accountability and governance requirement. However, it is remarkably difficult to facilitate or even know how to enable effective agent oversight in practice [7, 45, 52, 94, 112, 127]. As Section 2 explains, most research on oversight is conceptual or narrowly focused on specific tasks, such as finding mistakes in AI outputs. Even with the pressing need to examine on-the-ground practices, there is little empirical research that holistically examines how users actually oversee AI or agentic systems in messy real-world settings, rendering current oversight efforts largely aspirational [69, 94, 133, 136].

Related work. Although we focus on human oversight of novel software agents, our conceptualization of oversight draws from research on human oversight of conventional AI systems such as predictive algorithms and recommender systems [16, 29, 38, 52, 64, 65, 105]. While a detailed overview of that research is outside the scope of this paper (we do engage with FAccT research on AI oversight in subsection 2.3), two insights are helpful to contextualize this paper’s findings and arguments.

The first insight is that reviewing AI outputs is considered key to overseeing any AI system. Researchers debate the specifics of what constitutes–or should constitute–oversight, but agree that spotting and correcting mistakes is a crucial part of it [13, 70, 91, 94, 114]. The second insight is that the field of human oversight of AI has significant empirical gaps. Most discourse on oversight of AI, and now agentic AI, either has a narrow empirical focus on controlled studies of mostly artificial verification tasks using proxy measures (unsurprising, given the importance attributed to the oversight work of reviewing AI outputs) [e.g.: 12, 44, 46, 63, 72, 107] or is largely conceptual, focusing on high-level arguments, frameworks, and theories of oversight [e.g.: 29, 41, 65, 68, 70, 112, 127, 132].

The overall result is a disconnect between the research and practice of human oversight. Empirical research is only beginning to explore how users actually oversee AI systems in real-world settings. Without this knowledge, it is difficult to identify on-the-ground oversight work, practices, and challenges, let alone build effective mechanisms to support oversight in practice. We take steps to bridge this gap by providing exploratory qualitative analyses of the emergent oversight work, challenges, and heuristics of developers using software agents.

The 2021 breakthrough demonstrating the ability of large language models (LLMs) to analyze and generate software code [21] sparked widespread interest, driving significant technical advancements [116] including agentic frameworks such as OpenHands CodeAct [137]. Coding agents have since been applied to a wide range of software engineering tasks [54, 76], such as debugging and patching [9, 153], code translation [37, 144], code refactoring [53, 130, 147], and generation tests for bug reproduction or proof of vulnerability [2, 24, 83, 88]. Alongside technical advancements, developer productivity studies have demonstrated the positive impact of software agents on task time [92, 100]. Sarkar’s [106] natural experiment analyses measured another productivity metric, repository merge rates, and found that organizations using Cursor’s agent achieved 26% more merges than those that did not use it.

Method. We followed a criterion-sampling strategy based on experience using software agents for professional work. In this study, we define a software agent as one that autonomously plans and acts to accomplish user goals by using tools and interacting with the environment.1 Our recruitment materials included examples, such as Claude Code and Cline, to clarify what we meant. We did not consider an AI that just auto-completes code as an agent.

We first reached out to a few professional contacts, and employed snowball sampling to identify additional participants.

Participants filled out a screener to determine participation eligibility. We assessed our sampling criteria through a combination of participants’ self-reported agent experience and agent usage for professional work. We only selected those who were somewhat experienced with agents and used them at least weekly for professional work. Sixteen of 17 participants used agents several times a week for professional work (12 daily). Nine participants reported being ‘experienced’ with agents, four ‘somewhat experienced,’ and four ‘very experienced.’2 See Appendix A for further participant details.

Ours is also an intensity sample [98], concentrating on experienced developers who likely understand agentic AI better than people in other professions. Thirteen participants had seven or more years of programming experience, two had over four years, and two had more than a year, spanning domains such as ad experimentation, financial technology, research, and storage systems. Challenges seasoned developers experience are likely worse for practitioners with less 6 Dhanorkar, Passi & Vorvoreanu accurate mental models of agentic AI. We also specifically focused on power users in well-resourced organizations, as early adopters and heavy users are well-positioned to signal emergent practices and help model best practices.

Twelve of 17 participants work at the same large-scale tech organization as the authors. This occurred unintentionally We used a semi-structured approach for interviews. Rather than present a preconceived definition of oversight, we let participants explain what agent oversight meant to them and how they conducted it. We first asked developers to describe what they used agents for (see Appendix B for agent usage details). We then dived into how developers worked with agents, evaluated agents’ work (e.g., spotting mistakes in agent outputs), and the difficulties they faced in doing so. The structure of our findings (i.e., four forms of oversight work) is shaped by our analytical lens (i.e., explicitly asking users what they do before, during, and after agent execution). We deliberately used temporal anchoring to ground participant responses in end-to-end agent workflows. The structure of our findings is thus best seen as a useful organizing device and not an exact representation. Note, however, that participants did organize their responses temporally even without prompting when differentiating between the oversight work of co-planning and real-time monitoring. See Appendix C for details on interview questions.

Interviews were conducted 1:1 by the first author via video conferencing software, video-recorded, and transcribed using Marvin [78]. Participants completed a consent form prior to the interview. Interviews were scheduled for one hour, lasting 60 minutes on average, and conducted between July-August 2025. Participants received a US $40 gift card or equivalent for participation. The research study was approved by our organization’s institutional review board.

We followed a qualitative inductive analysis process guided by our research questions and in concert with existing literature [131]. Analysis proceeded in three rounds. In round one, the first author conducted open coding, creating a list of developer actions, practices, and challenges through in-vivo codes directly using participant quotes (e.g., “set deny lists” and “co-plan with agents”). In round two, the first author captured common themes across open codes, creating 13 axial codes (e.g., co-planning and review strategies).

Discussion. 5.1 Oversight entails not only reviewing and monitoring, but also controlling and steering Our findings reveal that oversight work is likely not only reactive and evaluative, but also preventative and proactive. Post hoc review is the most visible form of oversight work in our study and in research [13, 70, 91, 94]. Additionally, although our participants did little real-time monitoring, it is seen as a growing challenge for agent oversight [18, 50, 94, 135].

However, participants described that oversight involves more than just reviewing agent outputs and monitoring agent actions. Developers also control and steer agents. Oversight work begins—as a priori control—even before prompting. For instance, developers anticipate failures, preemptively configuring agents to prevent them from making specific mistakes.

Oversight work is also underway—as co-planning—before agents start executing tasks. For instance, developers revise plans with agents and seed solutions to them, pushing agents in desirable directions away from pitfalls.

Some of the oversight work we found fits traditional notions of oversight (e.g., reviewing and monitoring), while some (e.g., preemptive configuration) appear less like oversight and more like “good” software engineering practices.

However, participants described doing such anticipatory “pre” work at times with explicit oversight intentions (e.g., preventing known mistakes or failures from occurring altogether). This is not to say that our findings warrant redefining oversight to now include software engineering practices. Instead, surfacing the oversight aspirations underpinning control and steering work, we highlight that oversight work is emerging not just in expected spaces, but also elsewhere in prompts and settings.

We show different oversight work occurring throughout human-agent interaction, expanding research scholarship on what users do as humans-in/on/over-loops (e.g., we show that users anticipate, and not just react) and where oversight loops begin/end (e.g., we show that they begin long before prompting) [25, 29, 46, 135, 158]. Our insights into post hoc review and real-time monitoring extend existing research on transparency-driven approaches to human-AI and human-agent collaboration [50, 104, 127]. Our insights into a priori control and co-planning surface new research sites such as the role of “anticipation” [26, 35, 126], the affordances and limits of natural language-based human-agent communication [7, 77, 94, 119], and the “managerial” work users must do when using agents (and its impact on craftsmanship, learning, and professional identity) [31, 132, 143].

5.2 Oversight functions not through exhaustive awareness, but through practical heuristics As explained in subsection 2.1, existing research largely aspires for “effective” oversight by facilitating continuous, “meaningful” human control over AI and agents [16, 29, 105, 114, 117]. However, our findings show that such aspirations often prove unscalable and are decoupled from the practical realities of users running software agents. We found that during post hoc review participants opted for good-enough supervision driven by “bounded rationality” [121] rather than exhaustive awareness of agentic outputs and workings. Faced with complex agentic systems, developers adopt emergent heuristics that prioritize functional efficiency over idealized perfection. Plans become proxies for agentic working, test results stand in for code correctness, and skimming becomes the main information processing mechanism.

Conclusion. Our study provides an exploratory analyses of how software developers oversee coding agents in practice. Developers perform diverse oversight work using specific practices to achieve different goals. We identify pain points developers encounter when supervising agents and the emergent heuristics they adopt to navigate the intrinsic uncertainty of working with agents. Our findings reveal the nuanced ways users have begun exercising oversight, beginning as early as during configuration and planning, and continuing through ongoing verification. Doing oversight is less of a single act and more of a coordinated set of ad-hoc evolving practices. By starting to make these activities visible, we contribute early empirical anchors for understanding, researching, and improving agent oversight. Our work also reflects on broader implications for tooling, individuals, organizations and future research to study, design, and support oversight in practice to enable effective collaboration and complementarity between humans and agents.

Limitations. This research takes important first steps and provides valuable contributions on agent oversight in practice but is not without limitations. First, while we tried recruiting participants from diverse professional settings, most participants worked in well-resourced technology firms (e.g., generous token limits, rich tool integrations, and early access to new features). Monetary and computational costs mattered little in their oversight decisions. Future work must examine oversight challenges and implications in resource-constrained settings. Second, 12 of our 17 participants worked at the same technology company. Although their experiences were echoed by the 5 participants from other firms, one company’s over-representation likely introduced some bias in our findings. Future research on oversight in diverse organizational contexts can complement this research with a more holistic view of agent oversight work. Third, our participants delegated relatively small tasks to agents, rendering, for example, real-time monitoring optional and post hoc review easier. Future research must analyze agent oversight work in the context of complex, longer-running tasks, while also examining how agent builders conceptualize and facilitate these more demanding forms of oversight work and requirements.