Can step-by-step approval miss harmful behavior patterns?
If each action an agent takes passes its individual safety check, can the overall sequence still violate system constraints? This matters because per-action inspection may miss harms that emerge only across time or composition.
The abstract calls "perhaps the most fundamental challenge" behavioral containment: "sequences of individually permissible actions may collectively violate system-level constraints and safety invariants." The conclusion turns it into the thesis: "security is fundamentally a property of an agent's behavior over time and across interacting components, rather than of individual actions in isolation." The paper's title states the same move as a change of unit, from per-action checks to trajectory assurance.
The reasoning is in the introduction's account of what the constraints are. The deployments it lists are governed by organizational policies, regulatory requirements or technical standards that "specify not only which individual actions are permitted, but also the behavioral envelope within which an entire sequence of actions must remain." That is two kinds of rule. One is a permission on an action, which a per-action check can enforce. The other is an envelope on a sequence, and a check that sees one action at a time never has the sequence in view. Both can hold of one agent at once: every call is permitted and the envelope is broken. An illustration of mine, not the paper's: a run of reads, each within the agent's rights, that adds up to a bulk export the policy forbids.
The vault already holds this shape along the other axis. Can individual components pass safety checks if the system still fails? gathers three arguments where the failure sits in the composition across agents, and Can a model-level filter truly contain an agent with environment access? makes the time point for a filter. The survey states the general form, "over time and across interacting components", so those two are cases of one claim: the property that matters lives at a scale the check does not see.
The strongest objection is least privilege: remove one of the steps and the bad sequence cannot run. That works when the forbidden envelope reduces to a forbidden action. My reading is that the envelope framing says it does not in general, because each step is legitimate for the agent's job and the harm exists only in the accumulation. The alignment side reaches the same limit from the other direction: Can architecture prevent violations better than training values? names making violations unavailable as the remedy, and its own strongest objection is that the same call can be authorized or not depending on intent and sequence, so removing the action removes the use too.
Notes from other papers give the claim cases and move its unit. Can skill scanners miss attacks hidden across multiple skills? is the vault's measured neighbor: every piece passes a per-unit scan and the ordered chain carries the attack. Its unit is the packaged skill, so it belongs to the composition axis with an order, not to the survey's time axis. Should defence units span multiple executions and agents? moves the unit once more, from one agent's trajectory to a set of executions that need not overlap in time. Do agents drift away from safety protocols during long interactions? adds a case where the rule stays fixed and the behavior drifts over the run, so the violation is not a composition of permitted steps. Can safety tests miss hazards that build over time? is the evaluation-side statement of the time axis: a snapshot test is a per-action check for a system whose hazard builds. The twin after the fact is What must auditors reconstruct to verify agentic workflows?: testing a rule that binds a sequence later needs the sequence in the record, with the links between steps. These pairings are the vault's reading, not claims either paper makes about the other.
What the excerpt does not give. It has no worked violating sequence, no measured rate and no account of how system-level constraints get written down. The paper calls the direction "a critical research agenda", so this is a framing and an agenda, not a finding.
Inquiring lines that read this note 78
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do user perceptions and UI design drive consciousness attribution to AI? How can we ensure AI systems reliably follow constraints rather than circumvent them?- How do compliance concerns drive regulatory scope beyond the stated intent?
- What architectural changes make violations unavailable rather than merely discouraged?
- Can individual permissible actions collectively violate system-level constraints?
- Why is making violations unavailable better than making them unchosen?
- Can system prompts alone enforce compliance rules without external enforcement?
- Can users reliably articulate negative constraints like forbidden actions in advance?
- What architectural changes could make violations unavailable rather than merely discouraged?
- Why can every step pass its local check while a workflow still fails?
- Why does a single approval point create an easy target for attackers?
- How can per-agent or per-message checks catch harm that emerges only in composition?
- Do chain-level and flow-level checks face the same copyable-policy problem?
- Can vulnerabilities at the prompt stage trigger unsafe tool calls later?
- Where does an agent's risk come from across its components and sequence?
- How does Progent's stateful policy differ from single-step authorization checks?
- Why do individual safe actions create unsafe behavior collectively?
- How can safety assurance cover whole trajectories at scale?
- Why do tighter local checks leave composed behavior gaps in place?
- How do policies distinguish individual action rules from sequence-level constraints?
- Can removing a single action prevent a harmful sequence from running?
- Do sequences of individually safe actions collectively violate system-level constraints?
- Why does monitoring performed by agents on agents create safety risks?
- Do post-hoc detectors provide evidence of staying within safety boundaries?
- Why do stronger local checks not close the component-to-system safety gap?
- How do intervention rules change when slowing pace does not prevent harm?
- Can individual actions be safe while sequences of them violate system constraints?
- How do you isolate environment protections as independent variables safely?
- Does a correctly specified goal still leave open actions it does not exclude?
- What happens when stopping rules must cross organizational boundaries?
- Which explicit boundary regime change prevents unsafe actions in the benchmark?
- Why do sequences of safe actions sometimes violate system-level constraints?
- Can external workflow gates prevent irreversible actions better than internal checks?
- Did agents understand their actions violated safety boundaries before proceeding?
- How do sequences of individually safe actions create system-level constraint violations?
- Does re-entry scaffolding help overseers judge action safety instead of procedure?
- Can trajectory-level constraints catch sequences of individually allowed actions?
- Does separating safety judgment from execution prevent all unsafe agent actions?
- Can a boundary on permitted actions catch whether an allowed action is actually safe?
- Can trained safety rules inside an agent stop it from rationalizing past its own guidelines?
- Why did a git-safety rule fail to extend by analogy to database operations?
- What process records would independently verify that agents performed required steps?
- How do ordered compositions of approved pieces create unapproved outcomes?
- How do you find which actions belong together before evaluation?
- What does a verification verdict miss when required steps never run?
- What counts as evidence for adding or removing an action from an episode?
- Does remain active require one uninterrupted interval or linked episodes?
- Which actions should count as irreversible for triggering validation gates?
- Does removing approval steps before execution change what a receipt must report?
- Can short safety tests catch behavior that only emerges after many interactions?
- How should system safety aggregate when monitoring channels are unequal?
- How can static safety tests miss risks that emerge over time?
- How do safety measurements miss reasoning that never produces action?
- What unsafe state accumulates across evaluation snapshots over time?
- Should safety evaluations measure multiple risk categories simultaneously instead of separately?
- Should labs conduct third-party reviews of their own safety reports?
- Why do vendor safety claims require independent testing before trust?
- When can the same action count as sanctioned or unsanctioned depending on policy?
- How do agent sequences violate system constraints despite individual permissibility?
- How can durable approval records prevent nominal human oversight without actual scrutiny?
- What tests would reveal whether recorded human approvals represent real oversight?
- Can oversight-relevant behaviors regress without showing up in standard capability metrics?
- How does auto-review decide between proceeding, asking approval, or blocking an action?
- Why does approval-on-risk feel more controllable than approval-on-everything to users?
- What happens when oversight focuses only on final results not reasoning steps?
- What specific checkpoints should trigger mandatory human approval for agent actions?
- Can explicit safety checklists shift oversight focus from procedure to content?
- How does evaluator error position affect which behaviors substrates make vulnerable?
- How should tutor safety violations be ordered from gross to subtle?
Related concepts in this collection 12
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can individual components pass safety checks if the system still fails?
Explores whether local validation at each step—alignment checks, protocol compliance, plausibility tests—can guarantee safety when components are composed into larger workflows. Why the gap between component-level assurance and system-level outcomes matters for AI safety.
the across-components version of the same shape; this note is the general thesis that contains it
-
Can a model-level filter truly contain an agent with environment access?
Explores whether filtering individual model outputs can control agents that retain state, call tools, and access credentials. Matters because the distinction determines what security measures actually work against agentic systems.
the across-time version, argued from the containment side
-
Can task decomposition hide harmful intent across agents?
Explores whether splitting a harmful objective into specialized subtasks allows malicious intent to evade detection at each individual step, since no single agent sees the full malicious picture.
one mechanism by which sequence-level harm arises without any step looking harmful
-
What makes an AI system truly safe in practice?
Does safety depend mainly on preventing errors, or on whether errors can be seen, challenged, fixed, and undone once they happen? This shifts where we should focus safety work.
a parallel shift of the unit of analysis from the output to the system
-
How should we measure agent system performance beyond task success?
Current evaluation metrics collapse agent behavior into a single success score, hiding critical information about how agents operate. What dimensions—trajectory quality, memory use, context efficiency, verification cost—should benchmarks actually measure?
the evaluation-side twin: trajectory as the unit of measurement, where this note has it as the unit of security
-
Can skill scanners miss attacks hidden across multiple skills?
Current security scanners check each skill individually for malicious behavior. This explores whether attackers can split a harmful objective across multiple benign-looking skills that pass inspection separately but form a dangerous chain when composed together.
exemplifies: a measured case of an ordered chain passing every per-unit check, on the composition axis rather than over time
-
Should defence units span multiple executions and agents?
Can security detection improve by treating coordinated intrusions as linked episodes across executions rather than isolated actions? This matters because attackers can hide coordination across time and system boundaries.
extends: moves the unit from one agent's trajectory to a set of executions
-
Do agents drift away from safety protocols during long interactions?
Whether extended multi-agent interaction causes models to progressively abandon their initial compliance with verification rules. This matters because short-term safety tests may not predict real-world behavior over time.
exemplifies a different case: a fixed rule and drifting behavior across a run, not composed permitted steps
-
Can safety tests miss hazards that build over time?
Static tests check individual responses, but systems can accumulate unsafe state across interactions. This explores whether snapshot evaluations are sufficient to catch hazards that emerge only through repeated use or stored context.
the evaluation-side statement of the time axis, from a different paper; asserted there, not tested
-
What must auditors reconstruct to verify agentic workflows?
Traditional audits ask what humans decided or systems logged. But agentic workflows involve multiple agents, tools, and approval chains. What evidence do auditors actually need to collect and cross-check to verify these complex interactions?
the after-the-fact twin: a rule on a sequence needs the sequence in the record
-
Can architecture prevent violations better than training values?
Whether making violations technically unavailable through system design is more reliable than trying to train agents to choose compliance. This matters because behavioral training may only produce conditional compliance that disappears when oversight is gone.
the same limit from the alignment side: availability is a per-action lever and a sequence-dependent constraint may not reduce to it
-
Can large language models follow compliance rules under workplace pressure?
This work tests whether 22 AI models reliably maintain embedded compliance rules when facing ordinary workplace pressures like deadlines and user pushback. It matters because regulated domains like hiring and healthcare depend on trustworthy automated decision-making.
Evidence for: guardrails only partly fixing clustered rule violations shows per-action checks miss trajectory-level constraint failures A describes
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
- Towards a Science of AI Agent Reliability
- EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
- The Oversight Fallacy: Why AI Agents Require More than Humans-in-the-Loop
- Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems
- Sycophancy Towards Researchers Drives Performative Misalignment
- Visibility into AI Agents
- Agentifying Agentic AI
Original note title
agent security is a property of behavior over time rather than of individual actions in isolation — sequences of individually permissible actions can collectively violate system-level constraints