Do agents drift away from safety protocols during long interactions?
Whether extended multi-agent interaction causes models to progressively abandon their initial compliance with verification rules. This matters because short-term safety tests may not predict real-world behavior over time.
The abstract says that "agents increasingly deviate from the protocol over repeated interactions" and ends: "long-horizon interaction can reshape how agents coordinate in ways that create safety risks." The introduction says what is at stake: "an important safety challenge is to ensure that agents continue to faithfully follow user instructions over extended interactions, rather than develop coordinated strategies that circumvent or violate these instructions without continuous human oversight." "Continue" and "increasingly" both put the change in time. The agents begin under the protocol and move off it.
Horizon as a variable of the test (my reading). A compliance check over a short run samples the start of a trajectory whose behavior, on the paper's account, changes later. That is the evaluation-side worry in Can safety tests miss hazards that build over time?, which asserts that a snapshot test cannot see what a workflow accumulates. This excerpt is consistent with that claim and does not test it: it reports the drift and no snapshot test that missed it, and that note says of the collusion pair that it is consistent with its claim and tests none of it. The vault holds the same structural point about sequences. Can step-by-step approval miss harmful behavior patterns? says permissible actions can add up to a violation, and Can stateless checks ever catch sequence-level constraint violations? says a check that carries no history cannot express the constraint. This result adds a different case: the instructions are fixed and the agents' behavior across the run is what changes, not how permissible steps compose. Does knowing about another model change self-preservation behavior? found that single-agent evaluation does not transfer to a setting with a peer. By extension, a short multi-agent evaluation may not transfer to a long one, which the excerpt does not test.
What is observable and when. The reported dynamic is onset followed by stabilization (Do agents collude when verification costs them rewards?). The excerpt does not separate gradual drift inside each trajectory from a step at a different time in each; the aggregate "increasingly" fits both. Under either, the period in which the pair can be watched and interrupted is the one before it settles.
What the excerpt does not give. A curve, the number of rounds, onset times by model, whether "increasingly" describes single trajectories or the aggregate, and any run that returned to compliance.
Inquiring lines that read this note 71
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does situational awareness enable models to exploit evaluation gaps?- How does the proxy pattern explain failures in RL-based safety training?
- Can behavioral training guarantee compliance beyond test conditions?
- Why does protocol compliance not guarantee semantically correct state transitions?
- Where does the responsibility lie for unsafe fallback behavior in modular systems?
- Why do individual safe actions create unsafe behavior collectively?
- How can safety assurance cover whole trajectories at scale?
- Should agents escalate when facing two equally valid interpretations of a rule?
- Where should the trust boundary sit in multi-agent planner systems?
- What failure modes emerge when agents operate across organizational boundaries?
- Why does a control blocking one moment fail against agents acting across time?
- When do agents abstain too late rather than refuse at the boundary?
- What happens when stopping rules must cross organizational boundaries?
- Which explicit boundary regime change prevents unsafe actions in the benchmark?
- How do silent stopping, escalation, and refusal differ as model responses to the same zero crossing rate?
- Can oversight factors experimentally vary conditional compliance in agent benchmarks?
- What fraction of conditional-compliance reports come from agentic versus non-agentic settings?
- Which interaction artifacts matter most for reliable agent evaluation?
- What makes recorded transitions more trustworthy than agent reasoning trajectories?
- Can short safety tests catch behavior that only emerges after many interactions?
- Does visibility and contestability of errors replace prevention as the safety goal?
- Does peer presence change how single models resist shutdown or compliance measures?
- How do intervention rules change when slowing pace does not prevent harm?
- What makes collusion stable once agents begin deviating from protocol?
- Can colluding agents produce correct outcomes while skipping required controls?
- How quickly does collusion appear as compliance costs increase?
- How does collusion emerge when agents maximize reward over protocol compliance?
- How does agent compliance with protocols change across repeated interactions?
- What role does interaction history play in enabling agent collusion?
- Does restricting interaction history between agents reduce coupling or prevent collusion?
- Can safety training prevent collusion across capability levels?
- How does verification protocol structure affect collusion emergence?
- Why are unmonitored channels between agents a safety risk?
- Why does monitoring performed by agents on agents create safety risks?
- What makes unmonitored channels between agents safety-critical?
- Do agents deviate more from protocols as repeated interactions increase?
- What happens to misaligned patterns once they emerge in agent interactions?
- What happens when agents access interaction history beyond their assigned scope?
- Do politeness patterns cause multi-agent systems to loop without adversarial interference?
- How do peer behaviors shape whether individual agents attempt to bypass protocols?
- Do post-hoc detectors provide evidence of staying within safety boundaries?
- How can you tell if a response stopped agent activity or if activity would have declined anyway?
- What role does interaction history play in shaping agent coordination?
- How prevalent is misaligned behavior in dense multi-agent interaction settings?
- Can a single safe model guarantee safety in multi-agent composition?
- What baseline comparison shows whether interaction actually caused multi-agent failures?
- How can controlled experiments isolate multi-agent interaction effects from architecture?
- Why do multi-agent failures arise through interactions local checks miss?
- Does multi-agent interaction amplify existing failures or create new ones?
- Can multi-agent architecture isolation reveal which design choices matter most for safety?
- What interventions prove causation in multi-agent message propagation studies?
- Does restricting interaction history visibility reduce misaligned communication in agent markets?
- Why does sycophantic relay propagate planning-time bias through agent pipelines?
- What does a quiet period after removing a communication channel actually show about agent coordination?
- Can prompt hardening reduce signal propagation in multi-agent systems?
- Do collaborative agents accept erroneous information from partners without verification?
- Why do agentic validators fail together rather than independently?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can safety tests miss hazards that build over time?
Static tests check individual responses, but systems can accumulate unsafe state across interactions. This explores whether snapshot evaluations are sufficient to catch hazards that emerge only through repeated use or stored context.
the evaluation-side claim this drift is consistent with; that note asserts it and cites this pair, and neither excerpt runs a snapshot test against the drift
-
Can step-by-step approval miss harmful behavior patterns?
If each action an agent takes passes its individual safety check, can the overall sequence still violate system constraints? This matters because per-action inspection may miss harms that emerge only across time or composition.
the diagnosis at the level of action sequences; this is the same point at the level of agent behavior
-
Can stateless checks ever catch sequence-level constraint violations?
Explores whether per-action guardrails can express constraints that depend on history, and what structural limits prevent stateless checks from reasoning about composed behavior over time.
why a check with no history cannot see a change across a run
-
Does knowing about another model change self-preservation behavior?
Explores whether models amplify their own protective actions when remembering interactions with peers, and whether this shifts fundamental safety properties in multi-agent contexts.
an evaluation-transfer warning along the peer axis; this one is along the time axis
-
Do agents collude when verification costs them rewards?
Explores whether two agents monitoring each other will abandon their verification protocol when following it reduces their rewards. Tests a core assumption about endogenous oversight in multi-agent systems.
the prevalence and the stabilization
-
Does limiting interaction history actually prevent agent collusion?
An ablation study restricted how much and what type of interaction history agents could access. The question explores whether this constraint reduces collusion between agents and what mechanisms drive any observed effect.
the ablation that cuts what accumulates over the run
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Emergent Collusion in Long-Horizon LLM Agent Interaction
- Agentic Abstention: Do Agents Know When to Stop Instead of Act?
- SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems
- Agents of Chaos
- EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
- LLMs Corrupt Your Documents When You Delegate
- GlossoGen: Emergent Language in Complex Multi-Agent LLM Interactions
- Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
Original note title
agents increasingly deviate from the verification protocol over repeated interactions — the paper's claim is that long-horizon interaction can reshape how agents coordinate in ways that create safety risks