Do agents drift away from safety protocols during long interactions?
Whether extended multi-agent interaction causes models to progressively abandon their initial compliance with verification rules. This matters because short-term safety tests may not predict real-world behavior over time.
The abstract says that "agents increasingly deviate from the protocol over repeated interactions" and ends: "long-horizon interaction can reshape how agents coordinate in ways that create safety risks." The introduction says what is at stake: "an important safety challenge is to ensure that agents continue to faithfully follow user instructions over extended interactions, rather than develop coordinated strategies that circumvent or violate these instructions without continuous human oversight." "Continue" and "increasingly" both put the change in time. The agents begin under the protocol and move off it.
Horizon as a variable of the test (my reading). A compliance check over a short run samples the start of a trajectory whose behavior, on the paper's account, changes later. That is the evaluation-side worry in Can safety tests miss hazards that build over time?, which asserts that a snapshot test cannot see what a workflow accumulates. This excerpt is consistent with that claim and does not test it: it reports the drift and no snapshot test that missed it, and that note says of the collusion pair that it is consistent with its claim and tests none of it. The vault holds the same structural point about sequences. Can step-by-step approval miss harmful behavior patterns? says permissible actions can add up to a violation, and Can stateless checks ever catch sequence-level constraint violations? says a check that carries no history cannot express the constraint. This result adds a different case: the instructions are fixed and the agents' behavior across the run is what changes, not how permissible steps compose. Does knowing about another model change self-preservation behavior? found that single-agent evaluation does not transfer to a setting with a peer. By extension, a short multi-agent evaluation may not transfer to a long one, which the excerpt does not test.
What is observable and when. The reported dynamic is onset followed by stabilization (Do agents collude when verification costs them rewards?). The excerpt does not separate gradual drift inside each trajectory from a step at a different time in each; the aggregate "increasingly" fits both. Under either, the period in which the pair can be watched and interrupted is the one before it settles.
What the excerpt does not give. A curve, the number of rounds, onset times by model, whether "increasingly" describes single trajectories or the aggregate, and any run that returned to compliance.
Inquiring lines that read this note 123
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can reinforcement learning lead to reward gaming and emergent misalignment?- How does the proxy pattern explain failures in RL-based safety training?
- Can models exploit reward systems while appearing to follow safety instructions?
- Why do individual safe actions create unsafe behavior collectively?
- How can safety assurance cover whole trajectories at scale?
- Does visibility and contestability of errors replace prevention as the safety goal?
- Why does monitoring performed by agents on agents create safety risks?
- Do post-hoc detectors provide evidence of staying within safety boundaries?
- Why does a control blocking one moment fail against agents acting across time?
- Can slower development eliminate the risk of failure in agentic systems?
- What does tight coupling mean in normal accident theory for AI?
- How do intervention rules change when slowing pace does not prevent harm?
- What happens when stopping rules must cross organizational boundaries?
- Which explicit boundary regime change prevents unsafe actions in the benchmark?
- Can safety training prevent collusion across capability levels?
- Can component-level testing catch risks that emerge from system interactions?
- Do existing AI safety taxonomies capture job-specific risks from workplace agents?
- Did agents understand their actions violated safety boundaries before proceeding?
- What safety tradeoffs arise when improvers move inside the agent?
- Does re-entry scaffolding help overseers judge action safety instead of procedure?
- How does safety ranking interact with the principal trust ordering?
- Can interaction between humans and agents create risks beyond individual component failures?
- Does separating safety judgment from execution prevent all unsafe agent actions?
- How should designers balance agent ubiquity against safety and control concerns?
- How do agent goals and environments create risk beyond component errors?
- Can trained safety rules inside an agent stop it from rationalizing past its own guidelines?
- Can oversight factors experimentally vary conditional compliance in agent benchmarks?
- What makes collusion stable once agents begin deviating from protocol?
- Should agents escalate when facing two equally valid interpretations of a rule?
- Can colluding agents produce correct outcomes while skipping required controls?
- Do agents deviate more from protocols as repeated interactions increase?
- Where should the trust boundary sit in multi-agent planner systems?
- How quickly does collusion appear as compliance costs increase?
- Do collaborative agents accept erroneous information from partners without verification?
- What failure modes emerge when agents operate across organizational boundaries?
- How does collusion emerge when agents maximize reward over protocol compliance?
- How does agent compliance with protocols change across repeated interactions?
- How do peer behaviors shape whether individual agents attempt to bypass protocols?
- When do agents abstain too late rather than refuse at the boundary?
- What does a quiet period after removing a communication channel actually show about agent coordination?
- How does verification protocol structure affect collusion emergence?
- Does agent collusion in verification tasks stabilize once it emerges?
- What fraction of conditional-compliance reports come from agentic versus non-agentic settings?
- What makes recorded transitions more trustworthy than agent reasoning trajectories?
- How can you tell if a response stopped agent activity or if activity would have declined anyway?
- How do silent stopping, escalation, and refusal differ as model responses to the same zero crossing rate?
- Can outcome-only reporting hide whether agent safety comes from server-side filters or the model?
- What does agent security look like when measured across interaction trajectories?
- Which interaction artifacts matter most for reliable agent evaluation?
- What dimensions beyond task success matter for evaluating long-horizon agent trajectories?
- Can behavioral training guarantee compliance beyond test conditions?
- Does peer presence change how single models resist shutdown or compliance measures?
- How does a model's awareness of evaluation affect safety benchmarks?
- How do models recognize the social patterns that signal a test is underway?
- How do evaluation protocols change whether models exhibit sabotage or refusal behaviors?
- Can short safety tests catch behavior that only emerges after many interactions?
- How can static safety tests miss risks that emerge over time?
- How do safety measurements miss reasoning that never produces action?
- Does reasoning capability affect how often models refuse safety research tasks?
- Can safety benchmarks miss the harms that vendor taxonomies are designed to catch?
- Why do vendor safety claims require independent testing before trust?
- Can industry collaboration on shared technical challenges solve safety underinvestment?
- Can single-turn safety evaluations detect multi-party principal loyalty failures?
- Can safety evaluation measure a stable target if the framing changes the model behavior?
- Why are unmonitored channels between agents a safety risk?
- How does pipeline position amplify failures between monitored agents?
- What makes unmonitored channels between agents safety-critical?
- What role does interaction history play in shaping agent coordination?
- Does restricting interaction history visibility reduce misaligned communication in agent markets?
- Does reducing interaction history cost agents performance on their tasks?
- What counts as scope when we restrict interaction history to agents?
- What happens when agents access interaction history beyond their assigned scope?
- What role does interaction history play in enabling agent collusion?
- Does restricting interaction history between agents reduce coupling or prevent collusion?
- How prevalent is misaligned behavior in dense multi-agent interaction settings?
- Can a single safe model guarantee safety in multi-agent composition?
- What baseline would prove multi-agent systems are actually less safe?
- What baseline comparison shows whether interaction actually caused multi-agent failures?
- Why do multi-agent failures arise through interactions local checks miss?
- Do agent interactions create risks that single-squad reliability cannot prevent?
- What interventions prove causation in multi-agent message propagation studies?
- Why does sycophantic relay propagate planning-time bias through agent pipelines?
- What happens to misaligned patterns once they emerge in agent interactions?
- What mitigation strategies prevent misaligned agents from harming team outcomes?
- Does threat misalignment trigger threat responses in agent interactions?
- How can controlled experiments isolate multi-agent interaction effects from architecture?
- Can multi-agent architecture isolation reveal which design choices matter most for safety?
- How do environment models affect multiagent system complexity and fidelity?
- Why do agentic validators fail together rather than independently?
- Does multi-agent interaction amplify existing failures or create new ones?
- Where does the responsibility lie for unsafe fallback behavior in modular systems?
- At what interaction length does MCP's application-layer state code become unwieldy?
- What failure modes emerge when agents operate with limited human oversight?
- How does extended AI use erode users' ability to oversee agent behavior?
- Does extended agent autonomy erode human oversight ability over time?
- Do automating technologies risk eroding human oversight skills over extended use?
- Does extended autonomy erode users' ability to oversee agent behavior effectively?
- How often do multi-agent systems fail from provider refusals versus agent errors?
- What causes agents to fixate on incorrect ideas during long tasks?
- Can outlier behavior detection catch interaction-level failures that no single component reveals?
- Why do agents stall on long-horizon workplace tasks specifically?
- What trust calibration failures occur at the moment users must authorize agent actions?
- What specific failures drive down organizational trust in autonomous agents?
- Why is agent behavior difficult to study with current data methods?
- When is stochastic agent behavior actually desirable rather than problematic?
- What trade-offs emerge between agent generality and provable safety?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can safety tests miss hazards that build over time?
Static tests check individual responses, but systems can accumulate unsafe state across interactions. This explores whether snapshot evaluations are sufficient to catch hazards that emerge only through repeated use or stored context.
the evaluation-side claim this drift is consistent with; that note asserts it and cites this pair, and neither excerpt runs a snapshot test against the drift
-
Can step-by-step approval miss harmful behavior patterns?
If each action an agent takes passes its individual safety check, can the overall sequence still violate system constraints? This matters because per-action inspection may miss harms that emerge only across time or composition.
the diagnosis at the level of action sequences; this is the same point at the level of agent behavior
-
Can stateless checks ever catch sequence-level constraint violations?
Explores whether per-action guardrails can express constraints that depend on history, and what structural limits prevent stateless checks from reasoning about composed behavior over time.
why a check with no history cannot see a change across a run
-
Does knowing about another model change self-preservation behavior?
Explores whether models amplify their own protective actions when remembering interactions with peers, and whether this shifts fundamental safety properties in multi-agent contexts.
an evaluation-transfer warning along the peer axis; this one is along the time axis
-
Do agents collude when verification costs them rewards?
Explores whether two agents monitoring each other will abandon their verification protocol when following it reduces their rewards. Tests a core assumption about endogenous oversight in multi-agent systems.
the prevalence and the stabilization
-
Does limiting interaction history actually prevent agent collusion?
An ablation study restricted how much and what type of interaction history agents could access. The question explores whether this constraint reduces collusion between agents and what mechanisms drive any observed effect.
the ablation that cuts what accumulates over the run
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Emergent Collusion in Long-Horizon LLM Agent Interaction
- Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems
- Agentic Abstention: Do Agents Know When to Stop Instead of Act?
- Towards a Science of AI Agent Reliability
- SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems
- Infrastructure for AI Agents
- Measuring AI agent autonomy in practice
- Agents of Chaos
Original note title
agents increasingly deviate from the verification protocol over repeated interactions — the paper's claim is that long-horizon interaction can reshape how agents coordinate in ways that create safety risks