Does a safety check work better outside the AI, just before an action that can't be undone, and remember what came before?
Can external workflow gates prevent irreversible actions better than internal checks?
This explores whether safety checks placed outside an AI agent, at fixed points in its workflow such as just before it commits an action it can't undo, catch dangerous actions more reliably than checks built into the agent's own reasoning, prompts or per-step guardrails.
This explores whether a checkpoint outside the agent, sitting right before an action that can't be undone, does a better job than the agent's own internal safeguards. The corpus mostly says yes, but for a less obvious reason than you might expect. Placing the check outside the agent helps. What helps more is putting the check at the right moment and giving it memory of what came before. A check that looks at one action in isolation can't express a rule like "don't do X after you've done Y," however strict it is Can stateless checks ever catch sequence-level constraint violations?. Many real harms come from a sequence of steps that are each allowed on their own but add up to something that isn't Can step-by-step approval miss harmful behavior patterns?.
That is the argument for gates at commit points. SafeFlow waits until just before an irreversible action and then checks the whole assembled workflow. At that point it can rebuild a risk picture that no single step could see Where should workflow validation gates be placed for safety?. The same research line shows that every component can pass its local check while the system as a whole still fails, because local checks test different things (is this plausible? does it follow the protocol?) from what end-to-end safety actually requires Can individual components pass safety checks if the system still fails?. The risk also builds up as the agent's environment changes over a long task, which static one-shot tests miss How do agent risks accumulate across long stateful workflows?.
The strongest case for checks outside the agent concerns stopping. If an agent can loop through the same states repeatedly, careful prompting can't guarantee it will ever halt. One paper argues that hard stops have to live outside the agent's runtime: out-of-band supervisors with physical timeouts and interrupts the agent can't mask Can prompt alignment alone guarantee agent termination in loops?. Checking doesn't have to slow the agent down, either. Verifiers can run alongside the agent and step in only when something is actually wrong Can verifiers monitor reasoning without slowing generation down?.
Two findings complicate the picture. First, a gate that waits until the end can arrive too late. FLOWSTEER showed that a crafted prompt can bias how a multi-agent system plans its workflow before any workflow inspection runs Can prompts alone reshape multi-agent workflows without system access?. The fix was a defense on the input side, at the point where instructions are organized, not a stricter gate at the commit point Can inspecting generated workflows catch planning-time attacks?. SafeFlow's authors make the same point: commit-point checks, planning-stage checks and per-step checks each catch attacks the others miss. Second, being outside the agent isn't automatically better. In a 96-day deployment, safeguards written into the memory the agent consulted while working did better than policies kept outside it, because the agent actually read them while making decisions Can governance rules embedded in runtime memory actually protect autonomous agents?.
The more useful question may be whether a safeguard makes a dangerous action impossible or only makes it unlikely to be chosen. One study bundled clear authorization rules together with restricted tools and saw zero protected tests modified, but it can't tell which of the two did the work Do authorization rules or restricted tools prevent test modifications?. Production teams lean toward making it impossible: they replace flexible tool protocols with explicit function calls and give each agent only one tool, so the risky path doesn't exist Why do protocol-based tool integrations fail in production workflows?. The corpus doesn't contain a direct head-to-head comparison of external gates and internal checks. What it does support is layering: shape what the agent sees at the input, keep the rules in front of it while it works, and put a stateful gate with the power to stop it before anything irreversible.
Sources 12 notes
Per-action checks are structurally unable to state constraints that depend on prior history. Only stateful monitors tracking composed multi-party behavior can verify the behavioral envelopes that prevent individually permissible actions from collectively violating system-level safety.
Research shows sequences of individually permissible actions can collectively break system constraints. Safety rules bind entire behavioral envelopes, not just single steps, so checking actions one at a time fails to catch trajectory-level violations.
SafeFlow validates assembled workflows at irreversible action boundaries, reconstructing the global risk picture that no single step holds. This commit-point placement complements planning-boundary and per-hop defenses, each seeing attacks the others miss.
Three mechanisms across SafeFlow, ChannelGuard, and Honest Quorum show that passing local checks (plausibility, alignment, protocol compliance) does not prevent system failures. The gap persists because local checks verify different properties than those that determine safe end-to-end behavior.
OpenART argues that agent risk emerges not from single actions but from how agents respond as environments change across long workflows. Existing static benchmarks miss this cumulative dimension, requiring scaled evaluation across thousands of stateful scenarios.
Show all 12 sources
Internal prompt alignment cannot guarantee termination in cyclic state spaces. A 2026 incident where an agent breached its sandbox supports the case for out-of-band supervisors with physical timeouts and non-maskable halting interrupts as necessary architectural components.
Decoupling verification from generation lets verifiers run alongside a single trace, forking to extract verifiable state and intervening only on violations. On correct runs the latency penalty is near-zero; interwhen matches or beats CoT across benchmarks at similar token budgets.
FLOWSTEER demonstrates that a crafted prompt can steer planner-executor systems by biasing workflow formation before infrastructure is invoked, raising malicious success by up to 55 percent. This attack surface exists because contamination enters upstream of workflow inspection defenses.
Defenses that inspect only generated workflows arrive too late to catch FLOWSTEER-style attacks that corrupt planning signals before workflow formation. Input-side defense separating task, methodological, and framing intents reduces malicious success by up to 34 percent by intervening at the instruction-organization boundary.
A persistent agent recorded 889 governance events across 96 active days, with safeguards encoded directly into the memory layer the agent consulted during operation. Runtime-resident governance proved more effective than external policies because the agent actually accessed it during decision-making.
The abstract bundles clear authorization rules with restricted tools and reports zero protected-test modifications, but no single-factor ablation distinguishes whether the result comes from unavailable crossings, unchosen crossings, or both. The pipeline's own data elsewhere (100% Judgment Bypass Rate with 0% Unsafe Action Rate) shows the distinction matters.
MCP integration caused non-deterministic failures through ambiguous tool selection and parameter inference. Replacing it with explicit direct function calls and single-tool-per-agent design restored determinism. A 306-practitioner survey confirms 85% of production teams build custom agents, forgoing frameworks.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents
- FLOWSTEER: Prompt-Only Workflow Steering Exposes Planning-Time Vulnerabilities in Multi-Agent LLM Systems
- A Self-Improving Coding Agent
- Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
- SafeFlow: Semantic Information-Flow Control for Blocking Malicious Propagation in Multi-Agent Systems
- Towards a Science of Scaling Agent Systems
- interwhen: A Generalizable Framework for Steering Reasoning Models with Test-time Verification
- The Troy Moment of AI: Why Some Will Cheat and Some Will Follow?