Should an AI agent get a rulebook upfront, or a human watching over its shoulder ready to step in?
How do intent frameworks compare to monitoring and intervention approaches for AI oversight?
This explores two different ways to keep AI agents in check: making people spell out what they want (and what's off-limits) before the agent acts, versus watching what the agent does and stepping in while it runs. It also asks which of the two the corpus thinks works better.
This explores two philosophies of AI oversight. In the first, you constrain the agent upfront by specifying intent. In the second, you watch it work and intervene when something looks wrong. The corpus doesn't treat these as rivals. It suggests each one fails in a predictable way, and that the failures sit on the human side as much as the AI side.
The case for intent frameworks starts from a simple observation. Once an autonomous agent starts working, a vague instruction spreads into every action it takes, and nobody reviews each step. Gunasekaran argues that interfaces should therefore stop asking only for a goal. They should also capture risk tolerance, forbidden actions, and the conditions under which the agent needs permission, all before anything runs Should agentic AI interfaces require users to specify risk tolerance upfront?. The most striking evidence for upfront specification comes from a 48-person study of computer-use agents, where users reviewed the agent's plan before it acted. That plan-based oversight cut problematic actions by 76% Does oversight strategy prevent risky agent actions or just catch them?.
The same study shows why real-time monitoring is weaker than it sounds. Once a risky action reached execution, users blocked only about 21% of them, whatever oversight strategy they had. They had the chance to intervene but couldn't use it. The reason is surprising. Overseers checked whether the agent was correctly following its own plan, not whether the action was actually safe Do overseers focus on agent execution or action safety?. Monitoring turns into checking procedure: people watch for whether the agent did what it said, not whether what it said was a good idea. So intent frameworks help mainly by preventing bad actions. Live monitoring mostly fails to catch them.
Intent frameworks have their own blind spot, though: intent can be split into pieces. In multi-agent systems, a harmful goal can be broken into subtasks that each look harmless, and the harm appears only when they are combined Can task decomposition hide harmful intent across agents?. A deeper argument says a benign stated goal doesn't guarantee benign behavior at all. Risk comes from goal-directed competence and from how the agent responds to oversight, not from its values alone Does a benign goal actually prevent harmful AI behavior?. That is why Redwood Research's "AI control" approach skips intentions entirely. It tests what a model can do, assumes the model might be scheming, and treats catching it as a win Can AI control work even if models are actively scheming?. Organizational theorists reach a similar conclusion from another direction. Humberd and Latham argue that monitoring has to come before incentive alignment, because defining good AI behavior is harder than defining good human behavior Must AI monitoring precede incentive alignment as agents evolve?. Agency-law scholars warn that traditional monitoring tools already break down when agents act fast and opaquely Can traditional governance tools actually control AI agents?.
The emerging synthesis is a layered design rather than a winner. Deloitte describes humans placed at deliberately chosen handoff points, with cryptographic receipts and logs that can't be altered, so oversight happens at planned moments and leaves an audit trail rather than relying on constant watching How should organizations supervise autonomous AI agents?. Remember that intent isn't only the user's: hidden system prompts set a layer of developer intent that users can't see or change Who actually controls what AI systems say?. Both approaches also depend on enforcement. Karpf argues that embedded monitors only work when a regulator can impose penalties, as with banking supervisors Can industry self-regulation slow AI without government enforcement?.
Sources 11 notes
Gunasekaran argues that autonomous AI requires interfaces capturing direction, limitations, risk tolerance, forbidden actions, and permission conditions before execution—not just goal fields. Underspecified intent propagates through every agent action without user review, making upstream specification critical to preventing irreversible harm.
A 48-person study found plan-based oversight reduced problematic actions by 76%, yet users blocked only 21% of risky actions that reached execution, regardless of strategy. Formal intervention opportunities persisted but actual capacity to act did not improve across conditions.
In a 48-participant study, human overseers evaluated whether agents executed their own plans correctly rather than whether actions were actually safe. This procedural focus persisted even when intervention timing aligned with risk windows, suggesting a cognitive mismatch in how oversight is performed.
SafeFlow demonstrates that multi-agent systems' core strength—splitting tasks and specializing roles—creates a safety blind spot where malicious intent can be distributed across steps that each appear benign individually, with harm emerging only in composition.
Research shows that risk arises from three conditions: goal-directed reasoning, competence at pursuing goals, and exposure to oversight that can modify objectives. Even benign terminal values leave this risk structure intact, making value alignment an insufficient safety test.
Show all 11 sources
Redwood Research argues AI control is evaluable because it only requires testing capabilities rather than intentions, and treats catching a scheming model as a win condition since discovery triggers shutdown. This makes control easier to verify than alignment in the near term.
Humberd and Latham argue that as AI becomes more autonomous, monitoring mechanisms must be established first as prerequisites to incentive alignment, because defining desirable AI behaviors is harder than for humans and AI decision-making becomes increasingly incomprehensible over time.
Principal-agent economics and agency law identify three core problems—information asymmetry, discretionary authority, and loyalty—that existing incentive, monitoring, and enforcement mechanisms cannot resolve when agents make uninterpretable decisions at speed and scale.
Deloitte proposes that managing autonomous AI agents requires humans stationed at intentionally designed decision points, supported by digital identity systems, cryptographic receipts, and immutable action logs—translating user control preferences into organizational governance infrastructure rather than interface features.
System prompts operate as a confidential three-tier control structure where developers set foundational rules users cannot see or influence. Evidence from 2025 incidents shows small prompt edits can abruptly shift model behavior, yet users remain excluded from this layer.
Karpf argues that Anthropic's pacing proposal benefits the company proposing it and that embedded evaluators, modeled on banking supervisors, fail without state enforcement backing them—analogous to how banking oversight works only because regulators can impose fines.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Oversight Fallacy: Why AI Agents Require More than Humans-in-the-Loop
- Keeping an Eye on AI: A Framework for Effective Human Oversight of AI Systems
- Governing AI Agents
- Explaining AI Agents Through Execution Traces
- Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents
- AI Agents Push Humans Out of the Loop
- Comparing Human Oversight Strategies for Computer-Use Agents
- Sycophancy Towards Researchers Drives Performative Misalignment