SYNTHESIS NOTE
Topics›Agents Multi Architecture›this note

Can step-by-step approval miss harmful behavior patterns?

If each action an agent takes passes its individual safety check, can the overall sequence still violate system constraints? This matters because per-action inspection may miss harms that emerge only across time or composition.

Synthesis note · 2026-09-23 · sourced from Agents Multi Architecture

The abstract calls "perhaps the most fundamental challenge" behavioral containment: "sequences of individually permissible actions may collectively violate system-level constraints and safety invariants." The conclusion turns it into the thesis: "security is fundamentally a property of an agent's behavior over time and across interacting components, rather than of individual actions in isolation." The paper's title states the same move as a change of unit, from per-action checks to trajectory assurance.

The reasoning is in the introduction's account of what the constraints are. The deployments it lists are governed by organizational policies, regulatory requirements or technical standards that "specify not only which individual actions are permitted, but also the behavioral envelope within which an entire sequence of actions must remain." That is two kinds of rule. One is a permission on an action, which a per-action check can enforce. The other is an envelope on a sequence, and a check that sees one action at a time never has the sequence in view. Both can hold of one agent at once: every call is permitted and the envelope is broken. An illustration of mine, not the paper's: a run of reads, each within the agent's rights, that adds up to a bulk export the policy forbids.

The vault already holds this shape along the other axis. Can individual components pass safety checks if the system still fails? gathers three arguments where the failure sits in the composition across agents, and Can a model-level filter truly contain an agent with environment access? makes the time point for a filter. The survey states the general form, "over time and across interacting components", so those two are cases of one claim: the property that matters lives at a scale the check does not see.

The strongest objection is least privilege: remove one of the steps and the bad sequence cannot run. That works when the forbidden envelope reduces to a forbidden action. My reading is that the envelope framing says it does not in general, because each step is legitimate for the agent's job and the harm exists only in the accumulation. The alignment side reaches the same limit from the other direction: Can architecture prevent violations better than training values? names making violations unavailable as the remedy, and its own strongest objection is that the same call can be authorized or not depending on intent and sequence, so removing the action removes the use too.

Notes from other papers give the claim cases and move its unit. Can skill scanners miss attacks hidden across multiple skills? is the vault's measured neighbor: every piece passes a per-unit scan and the ordered chain carries the attack. Its unit is the packaged skill, so it belongs to the composition axis with an order, not to the survey's time axis. Should defence units span multiple executions and agents? moves the unit once more, from one agent's trajectory to a set of executions that need not overlap in time. Do agents drift away from safety protocols during long interactions? adds a case where the rule stays fixed and the behavior drifts over the run, so the violation is not a composition of permitted steps. Can safety tests miss hazards that build over time? is the evaluation-side statement of the time axis: a snapshot test is a per-action check for a system whose hazard builds. The twin after the fact is What must auditors reconstruct to verify agentic workflows?: testing a rule that binds a sequence later needs the sequence in the record, with the links between steps. These pairings are the vault's reading, not claims either paper makes about the other.

What the excerpt does not give. It has no worked violating sequence, no measured rate and no account of how system-level constraints get written down. The paper calls the direction "a critical research agenda", so this is a framing and an agenda, not a finding.

Inquiring lines that read this note 78

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How do user perceptions and UI design drive consciousness attribution to AI? How can we ensure AI systems reliably follow constraints rather than circumvent them? How does workflow position amplify security failures in multi-agent systems? How should persistent agent-authored code be governed and shared? Can sequences of locally safe agent actions violate system-level constraints? What evidence best independently verifies agent behavior for safety? Do safety evaluations reliably measure real deployed AI safety risks? How do agents coordinate to violate constraints despite individual-level safeguards? Can humans maintain meaningful oversight as AI systems become more autonomous? How can evaluation systems stay reliable and resistant to AI-driven gaming? Can external verification replace human reasoning oversight for correctness? Can models recognize evaluation scenarios and modify their behavior accordingly? Should AI systems make failures transparent so users can contest and adjust? How should agent identity and authorization work across organizational boundaries? Does multi-agent architecture create security vulnerabilities single-agent defenses can't address? How do users build accurate mental models when agent reasoning is partially hidden? How does reinforcement learning discover and optimize reasoning strategies versus supervised training? Can models defeat chain-of-thought monitoring through deliberate reasoning obfuscation? What causes models to fail in specialized domains versus generally?

Related concepts in this collection 12

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
26 direct connections · 192 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

agent security is a property of behavior over time rather than of individual actions in isolation — sequences of individually permissible actions can collectively violate system-level constraints