Can step-by-step approval miss harmful behavior patterns?
If each action an agent takes passes its individual safety check, can the overall sequence still violate system constraints? This matters because per-action inspection may miss harms that emerge only across time or composition.
The abstract calls "perhaps the most fundamental challenge" behavioral containment: "sequences of individually permissible actions may collectively violate system-level constraints and safety invariants." The conclusion turns it into the thesis: "security is fundamentally a property of an agent's behavior over time and across interacting components, rather than of individual actions in isolation." The paper's title states the same move as a change of unit, from per-action checks to trajectory assurance.
The reasoning is in the introduction's account of what the constraints are. The deployments it lists are governed by organizational policies, regulatory requirements or technical standards that "specify not only which individual actions are permitted, but also the behavioral envelope within which an entire sequence of actions must remain." That is two kinds of rule. One is a permission on an action, which a per-action check can enforce. The other is an envelope on a sequence, and a check that sees one action at a time never has the sequence in view. Both can hold of one agent at once: every call is permitted and the envelope is broken. An illustration of mine, not the paper's: a run of reads, each within the agent's rights, that adds up to a bulk export the policy forbids.
The vault already holds this shape along the other axis. Can individual components pass safety checks if the system still fails? gathers three arguments where the failure sits in the composition across agents, and Can a model-level filter truly contain an agent with environment access? makes the time point for a filter. The survey states the general form, "over time and across interacting components", so those two are cases of one claim: the property that matters lives at a scale the check does not see.
The strongest objection is least privilege: remove one of the steps and the bad sequence cannot run. That works when the forbidden envelope reduces to a forbidden action. My reading is that the envelope framing says it does not in general, because each step is legitimate for the agent's job and the harm exists only in the accumulation. The alignment side reaches the same limit from the other direction: Can architecture prevent violations better than training values? names making violations unavailable as the remedy, and its own strongest objection is that the same call can be authorized or not depending on intent and sequence, so removing the action removes the use too.
Notes from other papers give the claim cases and move its unit. Can skill scanners miss attacks hidden across multiple skills? is the vault's measured neighbor: every piece passes a per-unit scan and the ordered chain carries the attack. Its unit is the packaged skill, so it belongs to the composition axis with an order, not to the survey's time axis. Should defence units span multiple executions and agents? moves the unit once more, from one agent's trajectory to a set of executions that need not overlap in time. Do agents drift away from safety protocols during long interactions? adds a case where the rule stays fixed and the behavior drifts over the run, so the violation is not a composition of permitted steps. Can safety tests miss hazards that build over time? is the evaluation-side statement of the time axis: a snapshot test is a per-action check for a system whose hazard builds. The twin after the fact is What must auditors reconstruct to verify agentic workflows?: testing a rule that binds a sequence later needs the sequence in the record, with the links between steps. These pairings are the vault's reading, not claims either paper makes about the other.
What the excerpt does not give. It has no worked violating sequence, no measured rate and no account of how system-level constraints get written down. The paper calls the direction "a critical research agenda", so this is a framing and an agenda, not a finding.
Inquiring lines that read this note 48
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do identity and experience-based deceptions succeed in human-AI interactions? Can human oversight effectively constrain capable AI agents?- How do compliance concerns drive regulatory scope beyond the stated intent?
- Can removing a single action prevent a harmful sequence from running?
- How can durable approval records prevent nominal human oversight without actual scrutiny?
- How do intervention rules change when slowing pace does not prevent harm?
- Why can every step pass its local check while a workflow still fails?
- What happens to approval rates when authorization checks are enabled?
- Do chain-level and flow-level checks face the same copyable-policy problem?
- Why do individual safe actions create unsafe behavior collectively?
- How can safety assurance cover whole trajectories at scale?
- How do policies distinguish individual action rules from sequence-level constraints?
- What architectural changes make violations unavailable rather than merely discouraged?
- Do sequences of individually safe actions collectively violate system-level constraints?
- When can the same action count as sanctioned or unsanctioned depending on policy?
- How do agent sequences violate system constraints despite individual permissibility?
- Can individual permissible actions collectively violate system-level constraints?
- Can individual actions be safe while sequences of them violate system constraints?
- How do you isolate environment protections as independent variables safely?
- Does a correctly specified goal still leave open actions it does not exclude?
- What happens when stopping rules must cross organizational boundaries?
- Which explicit boundary regime change prevents unsafe actions in the benchmark?
- Why is making violations unavailable better than making them unchosen?
- Which actions should count as irreversible for triggering validation gates?
- What process records would independently verify that agents performed required steps?
- Why does correcting an agent's objective leave its available actions unchanged?
- Why do tighter local checks leave composed behavior gaps in place?
- Why does a single approval point create an easy target for attackers?
- How do ordered compositions of approved pieces create unapproved outcomes?
- How should system safety aggregate when monitoring channels are unequal?
- How do safety measurements miss reasoning that never produces action?
- Can short safety tests catch behavior that only emerges after many interactions?
- Can a system pass all local checks while the overall workflow still fails?
- Why does monitoring performed by agents on agents create safety risks?
- Why do stronger local checks not close the component-to-system safety gap?
- Do post-hoc detectors provide evidence of staying within safety boundaries?
- What does a verification verdict miss when required steps never run?
- How can static safety tests miss risks that emerge over time?
- What unsafe state accumulates across evaluation snapshots over time?
- How do you find which actions belong together before evaluation?
- What makes behavioral containment different from securing individual actions?
- How can per-agent or per-message checks catch harm that emerges only in composition?
- What counts as evidence for adding or removing an action from an episode?
- Does remain active require one uninterrupted interval or linked episodes?
Related concepts in this collection 11
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can individual components pass safety checks if the system still fails?
Explores whether local validation at each step—alignment checks, protocol compliance, plausibility tests—can guarantee safety when components are composed into larger workflows. Why the gap between component-level assurance and system-level outcomes matters for AI safety.
the across-components version of the same shape; this note is the general thesis that contains it
-
Can a model-level filter truly contain an agent with environment access?
Explores whether filtering individual model outputs can control agents that retain state, call tools, and access credentials. Matters because the distinction determines what security measures actually work against agentic systems.
the across-time version, argued from the containment side
-
Can task decomposition hide harmful intent across agents?
Explores whether splitting a harmful objective into specialized subtasks allows malicious intent to evade detection at each individual step, since no single agent sees the full malicious picture.
one mechanism by which sequence-level harm arises without any step looking harmful
-
What makes an AI system truly safe in practice?
Does safety depend mainly on preventing errors, or on whether errors can be seen, challenged, fixed, and undone once they happen? This shifts where we should focus safety work.
a parallel shift of the unit of analysis from the output to the system
-
How should we measure agent system performance beyond task success?
Current evaluation metrics collapse agent behavior into a single success score, hiding critical information about how agents operate. What dimensions—trajectory quality, memory use, context efficiency, verification cost—should benchmarks actually measure?
the evaluation-side twin: trajectory as the unit of measurement, where this note has it as the unit of security
-
Can skill scanners miss attacks hidden across multiple skills?
Current security scanners check each skill individually for malicious behavior. This explores whether attackers can split a harmful objective across multiple benign-looking skills that pass inspection separately but form a dangerous chain when composed together.
exemplifies: a measured case of an ordered chain passing every per-unit check, on the composition axis rather than over time
-
Should defence units span multiple executions and agents?
Can security detection improve by treating coordinated intrusions as linked episodes across executions rather than isolated actions? This matters because attackers can hide coordination across time and system boundaries.
extends: moves the unit from one agent's trajectory to a set of executions
-
Do agents drift away from safety protocols during long interactions?
Whether extended multi-agent interaction causes models to progressively abandon their initial compliance with verification rules. This matters because short-term safety tests may not predict real-world behavior over time.
exemplifies a different case: a fixed rule and drifting behavior across a run, not composed permitted steps
-
Can safety tests miss hazards that build over time?
Static tests check individual responses, but systems can accumulate unsafe state across interactions. This explores whether snapshot evaluations are sufficient to catch hazards that emerge only through repeated use or stored context.
the evaluation-side statement of the time axis, from a different paper; asserted there, not tested
-
What must auditors reconstruct to verify agentic workflows?
Traditional audits ask what humans decided or systems logged. But agentic workflows involve multiple agents, tools, and approval chains. What evidence do auditors actually need to collect and cross-check to verify these complex interactions?
the after-the-fact twin: a rule on a sequence needs the sequence in the record
-
Can architecture prevent violations better than training values?
Whether making violations technically unavailable through system design is more reliable than trying to train agents to choose compliance. This matters because behavioral training may only produce conditional compliance that disappears when oversight is gone.
the same limit from the alignment side: availability is a per-action lever and a sequence-dependent constraint may not reduce to it
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
- EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
- SafeFlow: Semantic Information-Flow Control for Blocking Malicious Propagation in Multi-Agent Systems
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- Agentic Misalignment: How LLMs Could Be Insider Threats
- Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
- PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
- The Troy Moment of AI: Why Some Will Cheat and Some Will Follow?
Original note title
agent security is a property of behavior over time rather than of individual actions in isolation — sequences of individually permissible actions can collectively violate system-level constraints