Keeping an Eye on AI: A Framework for Effective Human Oversight of AI Systems

The use of Artificial Intelligence (AI) in high-risk, decision-making scenarios presents technical, safety, and normative challenges; problems that may only be ameliorated by human oversight. However, notions of human oversight lack a common foundational understanding: oversight architectures are not well defined, the roles involved remain unclear, and implementation steps are opaque. Hence, researchers and practitioners struggle to determine how to design, implement, and evaluate systems that enable effective human oversight. This paper advances a practical framework for effective human oversight of AI systems, based on a cross-disciplinary perspective that draws on insights from computer science, human-computer interaction, psychology, philosophy, and law. The core contributions are: (1) a foundational framework, with a working definition, architecture and processes for effective human oversight of AI systems; (2) an initial template for documenting oversight architectures and processes, applied to diverse domains; and (3) a synthesis of open research challenges that need to be considered in the emerging field of effective human oversight of AI systems.

Introduction. Artificial Intelligence (AI) is increasingly being deployed across domains, impacting industry, society, and private lives. AI-enabled technologies can and will deliver many positive impacts, but their benefits also come with risks (Cummings, 2024; Olteanu et al., 2025; Shelby et al., 2023). Harmful outcomes can occur in various ways. For example, model drift can degrade performance over time, technical problems can cause failures in unexpected situations or complete operational breakdown (e.g., during power outages) (Vela et al., 2022; Javed et al., 2024; Lasko et al., 2024)). AI systems also face difficult-to-detect security threats such as adversarial attacks (Tully et al., 2025; Brohi and Mastoi, 2025). Beyond technical concerns, normative risks emerge when system behaviour leads to unfair, discriminatory, degrading, or privacy-violating outcomes (Celi et al., 2022; Alderman et al., 2025; Murdoch, 2021; Karimian et al., 2022; Yang et al., 2024). In high-risk domains such as healthcare or critical infrastructure, such risks can have life-threatening consequences. Human oversight is often proposed as a safeguard against these risks, but what such oversight should entail is typically unspecified.

This paper aims to make the notion of human oversight concrete by proposing a framework for human oversight that mitigates the risks from AI-enabled technologies in high-risk use cases. In part, our work is a response to regulations that explicitly require human oversight of AI; for instance, Article 14 of the EU’s AI Act mandates that high-risk AI systems must be overseen by natural persons, as a way to mitigate residual risks that technical safeguards cannot eliminate (Cheong, 2024; McBride et al., 2014; De-Arteaga et al., 2020). In line with prior research, human oversight comprises monitoring AI systems, detecting inaccurate or inadequate outputs, recognizing failures, spotting malicious manipulation, making normative judgments about justice and fairness, and intervening when deemed necessary (Langer et al., 2024, 2026b; van Voorst, 2024; Zerilli et al., 2019; Green, 2022; Sterz et al., 2024; Enqvist, 2023; Laux, 2024). Yet the significance of human oversight extends beyond regulatory compliance and jurisdictional boundaries. Across diverse domains, it is increasingly regarded as a critical—and perhaps final—safeguard against the manifold risks posed by AI systems, and as a critical pillar in the safety and accountability infrastructure for responsible AI.

While the conditions for effective human oversight of AI are yet to be understood, it is evident that oversight can fail easily. Human overseers can fail to detect inaccurate outputs or fail to intervene effectively (Rieger et al., 2025; Vaccaro et al., 2024; Gaube et al., 2021). Even more concerning, human oversight itself can introduce inaccuracies and biases (Rieger et al., 2025; Green and Chen, 2019), especially when control mechanisms are insufficient (Sterz et al., 2024). In other scenarios, oversight may not be humanly possible Green (2022). Most critically, poorly-designed oversight can create the illusion of human control, when oversight structures give humans the responsibility, but not the agency, to prevent harm (Romeo and Conti, 2025; Sterz et al., 2024).

Given these challenges, many fundamental questions about human oversight need to be answered, including: What should the goals of human oversight be? What oversight activities need to be carried out? Who is qualified to perform this oversight? Under what conditions can it be effective and meaningful? And what happens when effective human oversight is impossible or infeasible? Some of these questions are not new; decades of research on human–automation interaction (Mosier and Manzey, 2019; Sheridan, 2021; Bainbridge, 1983) have addressed similar issues in aviation, nuclear power, industrial production, and autonomous weapons systems (McBride et al., 2014; Schwarz, 2021) (where a recent NATO report shows the depth of the discussions on human oversight and control Draper and van Diggelen (2020)). However, the scale, complexity, and opacity of today’s AI systems—including foundation models and agentic systems that act autonomously across multiple steps—present new challenges, requiring new forms of oversight based on interdisciplinary understandings (Cummings, 2023; Endsley, 2023; Naikar et al., 2023; Narayanan and Feigh, 2026). Prominent challenges for oversight of AI systems include (Langer et al., 2024; Pritchett, 2024): increased agency that can lead to new anomalies and, hence, new risks (Acharya et al., 2025); the probabilistic, opaque nature of AI systems results in new level of complexity, including inter-dependencies (e.g., in multi-agent systems (Hammond et al., 2025)); and changes in model effectiveness over time (e.g., model drift (van der Vorst et al., 2025).

Method. Human oversight can be viewed as a control and safety layer, ideally one that prevents unfavourable outcomes from AI systems, and critically embedded in a larger risk management architecture Rasmussen (1997) given that human oversight cannot address all risks. From this perspective, Human Oversight of an AI system can be defined as a:

Deliberate human activity with the goal of sufficiently mitigating risks in the use of an AI system for a specific task. This oversight, which may be technologically supported, often works within distributed organisational structures and processes that have to meet regulatory and economic requirements. It requires one or more people to monitor the operation and output of the AI system and to intervene when professional judgment, established criteria, or legal provisions indicate that action is required to mitigate risks arising from the system’s processes, outputs, or its context of use. Its effectiveness is assessed by the reduced likelihood of occurrence and severity of harms arising from the system operations.

This definition applies to decisions that are dynamic and time-critical (e.g., overseeing autonomous vehicles) and highstakes, deliberative (e.g., medical, judicial, safety Cummings, 2024; Olteanu et al., 2025; Shelby et al., 2023). Below, we briefly expand on some key considerations in this definition:

Deliberate Human oversight does not arise incidentally. It requires explicit assignment: individuals must be designated to oversight roles, trained accordingly, and embedded within organisational structures.

Distributed Human oversight may also be distributed across individuals, hierarchical layers, and organisations, which may often be a key requirement for effective oversight (Grote et al., 2024). An operator may monitor single outputs, a domain expert may review borderline cases, developers may analyse AI system behaviour over time, and a compliance officer may monitor communication between the frontline operators and developers. Tasks with clear goals and responsibilities need to be explicitly allocated to these parties.

Monitor Monitoring is the ongoing process of observing (potentially aggregated) system behaviour and outputs against relevant benchmarks, such as pre-defined performance thresholds, legal standards, or professional norms.

Intervene Intervention is the action taken when monitoring reveals risks: correcting an output, halting a process, flagging a case, escalating a decision, or re-training or re-designing a system. Crucially, the capacity to intervene is predicated on the overseer having both the understanding to recognize when action is needed and the control or causal influence to execute that action (Sterz et al., 2024).

Effective Effective human oversight “reduces occurrence and severity of harms”. This can be realised in three increasingly demanding ways: handover, error detection, and synergy. Handover involves humans taking over when the AI system fails, or reaches the boundary of its competence. This required sufficient situational awareness to ensure safe transfer of control (Endsley, 2023). Error Detection involves identifying inadequate system behaviour and deciding whether to override the AI or not (Langer et al., 2024; Rieger et al., 2025). Its effectiveness depends on the detectability of errors, overseer capability to identify them, and on the efficacy of available interventions. Synergy requires appropriate reliance—intervening on erroneous outputs while trusting well-functioning AI—, up to the point where human-AI performance exceeds either alone(Vaccaro et al., 2024; Gaube et al., 2026; Eckhardt et al., 2025)). Here, oversight reduces both AI-related risks and risks introduced by human intervention.

Enabling human oversight effectiveness. Establishing, ensuring, and evaluating the effectiveness of human oversight is a socio-technical challenge, requiring the integrative design of human, technological, and organisational factors (as argued by e.g. Cheong, 2024; Kaber, 2025; Naikar et al., 2023; Rieger et al., 2025; Sterz et al., 2024; Laux, 2024).

Discussion. 5 Challenges Facing Effective Human Oversight Although our framework provides a practical starting point for implementing effective human oversight of AI, it also exposes central challenges: operational challenges in the core monitoring and intervention activities; socio-technical challenges that consider the interactions between the human, the task and the organisational context; and governance challenges, concerning the limits and legitimacy of human oversight.

Effective oversight depends on humans being able to monitor AI systems and appropriately intervene when needed, activities that pose difficulties that go beyond those encountered in traditional human-automation interaction. Moreover, human overseers may require new forms of expertise to meaningfully enact their responsibilities.

Monitoring AI systems often involves interpreting probabilistic, multi-modal, and dynamically changing information signals rather than deterministic ones (Naveed et al., 2025). Signals may drift over time, emerge only at aggregate levels, or be distributed across system components (Naveed et al., 2025; Shankar et al., 2024; Protschky et al., 2025). A challenge, therefore, is determining which information signals should be provided and how they should be represented across different task layers, risk profiles, and oversight roles.

Intervening in AI systems can require actions at multiple layers, including model modifications, full retraining, changes to data pipelines, or the introduction of human-in-the-loop validation (Shankar et al., 2024; Bhargava and Singhal, 2024). Effective intervention, therefore, requires mapping how AI systems are developed, deployed, and maintained across data sources, supply chains, and model layers (Sambasivan et al., 2021; Balayn et al., 2025).

Research in human-automation interaction has identified persistent factors that impede human control (Kaber, 2025; Mosier and Manzey, 2019; Sheridan, 2021). Fundamentally, human oversight is constrained by well-documented limits of human cognition, including reliance on heuristics (i.e., bounded rationality, (Simon, 1990; Mosier and Manzey, 2019; Hollan et al., 2000)). These limits are compounded by the restricted effectiveness of explainability (Kaur et al., 2022; Miller, 2023; He et al., 2023; Rieger et al., 2023; Gaube et al., 2023, 2026; Cecil et al., 2024; Tintarev, 2025) and transparency (Liao et al., 2023; Hussain and Hussain, 2025), which do not guarantee meaningful understanding or reliable intervention. Oversight tasks also frequently involve prolonged periods of passive monitoring, a mode of engagement for which humans are poorly suited due to limitations in sustaining vigilance (Klein and Feltmate, 2025). These challenges may be further exacerbated by organisational goals and incentive structures that undermine effective oversight, such as productivity targets that reward rapid acceptance of AI outputs rather than critical engagement (Shankar et al., 2024; Parasuraman and Manzey, 2010).

Even where human oversight is feasible in principle, sustaining it in practice requires carefully designed sociotechnical conditions (Cummings, 2023; Grote, 2023; Naikar et al., 2023; Pritchett, 2024). Effective oversight depends on aligning interfaces, workflows, organisational structures, and training with the demands of monitoring and intervention. Oversight also increasingly requires AI-specific technical competencies, including understanding model behaviour and error modes (Naveed et al., 2025).

Conclusion. In this paper, we proposed a framework for effective human oversight of AI systems. Built on cross-disciplinary perspectives from participants at the 2025 Dagstuhl Seminar on human oversight, the key insights reflected in the framework are: (a) that human oversight of AI is a multi-layered issue, which goes beyond real-time intervention to include systemic and compliance layers; and (b) monitoring and interventions are required at these layers, and require new areas of expertise than we have seen with oversight of ‘traditional automation’. We provided a template for human oversight architectures and processes designed to engage stakeholders when reflecting on the socio-technical conditions for effective human oversight. We have shown this template can be applied in diverse domains, demonstrating the practical applicability of the framework.

We hope that this framework provides a shared foundation for researchers and practitioners working on human oversight of AI systems. However, it cannot resolve all challenges that arise as AI systems, especially systems with agentic capabilities that raise new practical, ethical, and legal demands for human control. We have, therefore, outlined key challenges for future research on human oversight, to focus the community’s research efforts. Many of the issues raised are not completely new, but they are arguably more complex than those seen before. Furthermore, the growing capabilities and widespread deployment of AI systems render these issues as urgent, and a major research program on effective human oversight of AI will be required to resolve them.