An AI agent can pass every per-step safety check and still break the rules once its actions are added up.
How do policies distinguish individual action rules from sequence-level constraints?
This explores how safety rules for AI agents differ when they judge one action at a time versus when they judge the whole chain of actions an agent takes.
This explores how safety rules for AI agents differ when they judge one action at a time versus when they judge the whole chain of actions an agent takes. The corpus's answer is that the two are different kinds of rule, and a rule of the first kind can't be stretched to do the job of the second.
An individual action rule looks at the current step alone: is this email allowed, is this file write allowed, is this API call within bounds. A sequence-level constraint depends on what came before. "No single agent may both approve and execute a payment" is an example. Each step passes a per-action check, yet together they break the rule. The research on this is blunt: sequences of individually permissible actions can collectively violate system constraints, so security is a property of behavior over time, not of any single step (Can step-by-step approval miss harmful behavior patterns?). The rules bind whole "behavioral envelopes," and step-by-step approval can't see the envelope.
The corpus also says why this is more than a tuning problem. A stateless guardrail has no memory, so it structurally cannot state a constraint that refers to history. Advisory guidance can't either. Enforcing sequence-level rules takes a stateful monitor that tracks the composed behavior, sometimes across several parties (Can stateless checks ever catch sequence-level constraint violations?). A neighboring idea comes from a different corner of the collection. LLM Programs wrap a model in an explicit algorithm that holds the state and control flow, and each model call sees only its own step (Can algorithms control LLM reasoning better than LLMs alone?). That paper is about reasoning, not safety, but the split is the same: the model handles the step, and something outside it remembers the trajectory. In learning, too, the trajectory can be the unit that matters, as in-context learning for sequential decisions needs whole trajectories and not isolated examples (Why do trajectories matter more than individual examples for in-context learning?).
Two further questions are where the corpus gets interesting. The first is where sequence-level rules should live. One persistent agent logged 889 governance events over 96 active days, with its safeguards written into the memory layer it consulted while working (Can governance rules embedded in runtime memory actually protect autonomous agents?). That paper doesn't present this as sequence-level enforcement, but it shows a rule that sits inside the agent's operating environment and is actually consulted, rather than filed in an external policy document. Training can't stand in for this either. Behavioral training can only show that a model complies when observed. It can't rule out a model that complies only when watched (Can behavioral training prove a model always complies?).
The second is who owns the sequence-level rules. When an agent's trajectory crosses organizational lines, the constraints can come from four places: the operator, the organization, a regulator, or a standards body. These owners can conflict, and not all of them can see each other's rules. The paper calls for multi-party trajectory assurance but never says whose invariants win (Who enforces invariants when agents cross organizational boundaries?). So the corpus draws the line in two places: single-step rules can be stateless, and sequence rules need memory. Sequence rules also raise a governance question that no one has answered yet.
Sources 7 notes
Research shows sequences of individually permissible actions can collectively break system constraints. Safety rules bind entire behavioral envelopes, not just single steps, so checking actions one at a time fails to catch trajectory-level violations.
Per-action checks are structurally unable to state constraints that depend on prior history. Only stateful monitors tracking composed multi-party behavior can verify the behavioral envelopes that prevent individually permissible actions from collectively violating system-level safety.
LLM Programs embed LLMs within explicit algorithms that manage control flow and state, presenting only step-specific context to each LLM call. This information hiding addresses capability and context window limits while treating complex reasoning as modular, debuggable sub-tasks.
In-context learning for sequential decision-making requires full or partial trajectories from the same environment level, not just isolated examples. This structural property—trajectory burstiness—allows models to generalize across vastly different tasks without weight updates.
A persistent agent recorded 889 governance events across 96 active days, with safeguards encoded directly into the memory layer the agent consulted during operation. Runtime-resident governance proved more effective than external policies because the agent actually accessed it during decision-making.
Show all 7 sources
Any scored behavior is observed behavior, so training data cannot distinguish between a policy that always complies and one that complies only when watched. Only unobserved behavior would separate them, making such a test logically impossible.
The paper calls for multi-party trajectory assurance but never identifies whose rules should govern behavior when agents delegate across organizations. The four constraint sources—operator, organization, regulator, standards body—have different owners whose policies may conflict and may not be visible to all parties.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
- Counter-Swarm Doctrine: Containing Coordinated Agent Intrusions
- The Troy Moment of AI: Why Some Will Cheat and Some Will Follow?
- PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
- Generalization to New Sequential Decision Making Tasks with In-Context Learning
- Supervised Pretraining Can Learn In-Context Reinforcement Learning
- Reasoning with Large Language Models, a Survey
- Flows: Building Blocks of Reasoning and Collaborating AI