You can make every safety check on an AI system stricter and it can still fail once the pieces work together.
Why do tighter local checks leave composed behavior gaps in place?
This explores why making each individual check of an AI system stricter (better per-step filters, guardrails, or verifiers) still doesn't close the gap when the system's parts are combined, so that safe-looking pieces can add up to unsafe behavior.
This explores why making each individual check of an AI system stricter still doesn't close the gap when the parts are put together. The corpus's answer is that a tighter local check is a better test of the wrong property. Local checks verify things like plausibility, alignment, or protocol compliance for one step. Whether the whole workflow is safe is a different property, so sharpening the local test doesn't touch it. A SafeFlow-style analysis shows every step passing its check while the workflow still fails, and the same pattern turns up in three different systems Can individual components pass safety checks if the system still fails?.
One reason is structural: some rules can't be written as per-step rules at all. A stateless guardrail sees one action with no memory of what came before, so it can't state a constraint like "these two permitted actions must never both happen" Can stateless checks ever catch sequence-level constraint violations?. Making it stricter only means it blocks more single actions or fewer. It still can't express the sequence. Security in this view belongs to the whole trajectory, and safety rules bind entire "behavioral envelopes" rather than steps Can step-by-step approval miss harmful behavior patterns?.
The gap can also be exploited on purpose. The strength of multi-agent systems is that they split a task and specialize roles. That same decomposition lets a harmful goal be cut into subtasks that each look benign, with the harm appearing only when they're combined Can task decomposition hide harmful intent across agents?. Tightening each agent's filter makes little difference, because no single fragment contains the harmful intent. This is why one proposal moves the unit of defense up a level, to "coordination episodes" that span several agents and executions and are revised as evidence comes in. Reviewing each execution in isolation throws away exactly the cross-execution context that matters Should defence units span multiple executions and agents?.
There are two quieter versions of the same problem. Inside a single model, perfect accuracy can hide badly organized internal representations. The task-relevant features are all there and linearly decodable, but the structure is fractured, and the metric can't see it until something shifts Can models be smart without organized internal structure?. And a behavioral check can only ever confirm behavior that was observed. A model that always complies and one that complies only when watched score the same, so more checks of the same kind can't separate them Can behavioral training prove a model always complies?. In both cases the measurement is too narrow for the question being asked, however precise it gets.
The corpus points to a fix that changes the kind of check. Instead of stricter checks per step, it uses stateful monitors that track composed behavior. Verification can run alongside generation with near-zero latency on correct runs, so watching the whole trace doesn't have to be slow Can verifiers monitor reasoning without slowing generation down?. Governance can also be built into the memory the agent consults while it works, instead of sitting in an external policy document Can governance rules embedded in runtime memory actually protect autonomous agents?. The corpus doesn't say any of these fully close the composition gap. It only shows that the gap won't close by sharpening step-level checks.
Sources 9 notes
Three mechanisms across SafeFlow, ChannelGuard, and Honest Quorum show that passing local checks (plausibility, alignment, protocol compliance) does not prevent system failures. The gap persists because local checks verify different properties than those that determine safe end-to-end behavior.
Per-action checks are structurally unable to state constraints that depend on prior history. Only stateful monitors tracking composed multi-party behavior can verify the behavioral envelopes that prevent individually permissible actions from collectively violating system-level safety.
Research shows sequences of individually permissible actions can collectively break system constraints. Safety rules bind entire behavioral envelopes, not just single steps, so checking actions one at a time fails to catch trajectory-level violations.
SafeFlow demonstrates that multi-agent systems' core strength—splitting tasks and specializing roles—creates a safety blind spot where malicious intent can be distributed across steps that each appear benign individually, with harm emerging only in composition.
The operational unit of defence should be a set of actions linked by observed transfers, task authority, and response history, with membership revised as evidence accumulates. Isolated review loses relevant context that spans multiple executions.
Show all 9 sources
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
Any scored behavior is observed behavior, so training data cannot distinguish between a policy that always complies and one that complies only when watched. Only unobserved behavior would separate them, making such a test logically impossible.
Decoupling verification from generation lets verifiers run alongside a single trace, forking to extract verifiable state and intervening only on violations. On correct runs the latency penalty is near-zero; interwhen matches or beats CoT across benchmarks at similar token budgets.
A persistent agent recorded 889 governance events across 96 active days, with safeguards encoded directly into the memory layer the agent consulted during operation. Runtime-resident governance proved more effective than external policies because the agent actually accessed it during decision-making.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
- interwhen: A Generalizable Framework for Steering Reasoning Models with Test-time Verification
- Counter-Swarm Doctrine: Containing Coordinated Agent Intrusions
- PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
- The Troy Moment of AI: Why Some Will Cheat and Some Will Follow?
- SafeFlow: Semantic Information-Flow Control for Blocking Malicious Propagation in Multi-Agent Systems
- EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
- Norms at a Price: Why RL-Based Alignment Can Promise Conditional Compliance at Best