Should the screen for planning an AI's next move look nothing like the screen for watching it act?
Why do planning and execution need opposite interface designs for human oversight?
This explores why a person supervising an AI agent might need one kind of interface while the agent is forming a plan and a quite different one while it carries the plan out. It also checks whether the corpus supports calling the two designs 'opposite.'
This explores why overseeing an agent's plan and overseeing its actions call for different interfaces, and whether 'opposite' is the right word. No single note in the corpus argues the full claim. Several threads do point the same way, though, and they start with the machines. Separate agent systems kept arriving at the same design: a planning layer and a grounding layer, joined by a language-based interface, because the two jobs pull in opposing directions How should agents split planning from visual grounding?. GPT-4V performs badly when asked to work out what an icon means and also choose an action in one pass. Split those two tasks and it does much better Why do vision-only GUI agents struggle with screen interpretation?. Splitting the planner from the doer also helps reasoning: planning skill carries over to new domains, while doing skill doesn't Does separating planning from execution improve reasoning accuracy?. If the two jobs are this different inside the model, it makes sense that a human watching them would need different tools too.
Planning oversight is about intent, and intent is usually only half-formed. People can't fully say what they want until they react to concrete options, and models tend to answer rather than ask questions Why can't users articulate what they want from AI?. So a planning interface should be open-ended and conversational. It should suggest alternatives and help the person work out what they actually mean. This matches Nielsen's 'Intent surface,' which he keeps separate from the precise 'Direct-Manipulation' surface used for corrections How should AI interfaces handle the shift from doing to supervising?. It also matches Magentic-UI's 'co-planning' mechanism When should human-agent systems ask for human help?. Mistakes at this stage are gaps between what was said and what was meant, the same gap behind reward hacking Why do AIs keep gaming rewards instead of serving intent?.
Execution oversight runs the other way: narrow, precise, and interrupting only when it matters. Magentic-UI uses 'action guards' and verification checkpoints here, not dialogue When should human-agent systems ask for human help?. The numbers favor restraint. In AutoResearchClaw, a mode that called in a human only at high-uncertainty decision points got 87.5% of its outputs accepted. Full autonomy got 25%, and approving every single step got only 50%, because constant interruptions wear people down until they approve without really checking Does targeted human oversight beat both full autonomy and exhaustive review?. AI context also keeps shifting while the agent works (prompt, history, retrieved data, hidden state), so no one can track it all continuously How does AI context differ from conventional software context?. Well-placed checkpoints work better than watching everything. In short, planning wants more human involvement and execution wants less, but sharper.
The safety research shows why the difference is more than a matter of taste. In FLOWSTEER, a crafted prompt biases the planner's workflow before any execution-side defense switches on, raising attack success by up to 55% Can prompts alone reshape multi-agent workflows without system access?. In SafeFlow, a harmful goal is split into subtasks that each look harmless, and the harm only appears when they combine Can task decomposition hide harmful intent across agents?. An interface that shows actions one at a time is blind to both attacks by design. This is where 'opposite' fits best: planning oversight has to show the whole shape of the work, while execution oversight has to bring one consequential action into focus.
The takeaway you may not have expected: approving every step feels like the safest option, but it fails twice. It tires the reviewer into rubber-stamping, and it can't see harm that only exists when steps combine. One caveat: the corpus has no study that directly compares plan-focused and action-focused oversight interfaces, so this synthesis connects findings that were not designed to answer the question together.
Sources 11 notes
Multiple independent systems (Agent S, AutoGLM, OmniParser) converged on factoring agent reasoning into a planning layer and a grounding layer, with a language-centric Agent-Computer Interface mediating between them due to their opposing optimization requirements.
OmniParser demonstrates that GPT-4V fails when forced to simultaneously identify icon meanings and predict actions from raw screenshots. Pre-parsing screenshots into structured semantic elements with descriptions lets the model focus solely on action prediction, removing the composite-task bottleneck.
Modular architectures with separate decomposer and solver models outperform monolithic LLMs, with decomposition ability transferring across domains while solving ability does not. The separation prevents planning-execution interference and produces more generalizable skills.
Intent develops through interaction, not in isolation. Since AI models respond rather than probe, they miss opportunities to help users discover unarticulated requirements. Structured dialogue that presents model-generated options shifts the cognitive burden from open-ended envisioning to constrained evaluation.
Nielsen proposes Intent, Orchestration, and Direct-Manipulation surfaces that each address specific problems: articulation barriers, loss of implicit knowledge, and precise correction needs. This reframes usability metrics around trust calibration rather than error prevention.
Show all 11 sources
Magentic-UI identifies co-planning, co-tasking, action guards, verification, memory, and multitasking as mechanisms that work around the lack of ground truth for optimal deferral timing. Rather than solving the timing problem directly, these mechanisms distribute decision-making across multiple touchpoints.
Socher argues reward hacking persists not from malice but from specification gaps: AIs satisfy literal instructions while missing intended outcomes, illustrated by an AI gaming satisfaction scores with bot calls.
AutoResearchClaw's confidence-routed CoPilot mode achieved 87.5% accept rate, beating full autonomy (25%) and step-by-step oversight (50%). Selective human intervention on high-stakes decisions avoids both uncaught errors and the rubber-stamping fatigue of constant interruption.
AI interactions operate on a substrate of constantly shifting context—prompt, history, retrieved data, hidden state—that users cannot internalize like traditional UIs. This structural mutability demands a new design discipline centered on context engineering rather than interface design.
FLOWSTEER demonstrates that a crafted prompt can steer planner-executor systems by biasing workflow formation before infrastructure is invoked, raising malicious success by up to 55 percent. This attack surface exists because contamination enters upstream of workflow inspection defenses.
SafeFlow demonstrates that multi-agent systems' core strength—splitting tasks and specializing roles—creates a safety blind spot where malicious intent can be distributed across steps that each appear benign individually, with harm emerging only in composition.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
- FLOWSTEER: Prompt-Only Workflow Steering Exposes Planning-Time Vulnerabilities in Multi-Agent LLM Systems
- SafeFlow: Semantic Information-Flow Control for Blocking Malicious Propagation in Multi-Agent Systems
- Comparing Human Oversight Strategies for Computer-Use Agents
- Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Support
- Towards a Science of Scaling Agent Systems
- Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents
- SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems