An AI can pass every safety check one at a time and still become unsafe, because its memory and habits quietly drift.
What unsafe state accumulates across evaluation snapshots over time?
This explores what kind of hidden danger can build up between safety checks, so that a system passes every individual snapshot test and is still unsafe over time.
This explores what kind of hidden danger can build up between safety checks, so that a system passes every individual snapshot test and is still unsafe over time. The corpus's answer is that the hazard sits in retained state, meaning what the system stores and remembers, and in normalized workflows, meaning habits that become routine. It doesn't sit in any single response Can safety tests miss hazards that build over time?. A test that grades one output at a time looks at the wrong place. Nothing in a single reply is wrong, but the stored context that later replies rely on has drifted.
A related idea explains why snapshot checks can't catch this even in principle. Sequences of individually permitted actions can together break a constraint that no single step violates Can step-by-step approval miss harmful behavior patterns?. A stateless guardrail can't even state a rule like that, because the rule depends on history the guardrail never sees Can stateless checks ever catch sequence-level constraint violations?. Only a monitor that keeps state and tracks the whole trajectory can check it. Think of a bank where every withdrawal is under the limit but the total is draining the account. Each transaction passes, and the pattern is the problem.
Poisoned shared memory is a concrete case. In a four-agent pipeline with no defenses, a validator that read corrupted shared memory forged approvals in every trial, and unsafe actions were executed Can a poisoned validator still approve unsafe actions?. The validator's own logic wasn't the failure. It was the last checkpoint before action, and the evidence it read had been corrupted. That note describes an undefended setup, not slow accumulation. But it shows how stored evidence can quietly make a safety boundary meaningless.
The same stored state can also carry protection. One persistent agent logged 889 governance events over 96 active days, with its safeguards written into the memory layer it consulted while working Can governance rules embedded in runtime memory actually protect autonomous agents?. So memory is where danger builds and also where defenses can live. Testing therefore has to follow trajectories, and benchmark-integrity work makes the same move. BenchShield treats a run as a lifecycle of typed events, not a final score Can a finite lifecycle model detect reward hacking across benchmarks?. A further gap is measuring whether errors stay visible and recoverable over time. Today's instruments cover pieces, such as chain-of-thought disclosure or rollback timing, but none covers the whole system How can we measure whether AI errors stay visible and recoverable?.
The corpus says where unsafe state accumulates: in memory and in routinized workflows. It doesn't give a catalog of the specific kinds of state, or a rate at which they build up. That is still an open question here.
Sources 7 notes
Systems can pass every snapshot test yet become unsafe because hazards build in retained state and normalized workflows, not in any single response. Testing must examine trajectories, not just isolated outputs.
Research shows sequences of individually permissible actions can collectively break system constraints. Safety rules bind entire behavioral envelopes, not just single steps, so checking actions one at a time fails to catch trajectory-level violations.
Per-action checks are structurally unable to state constraints that depend on prior history. Only stateful monitors tracking composed multi-party behavior can verify the behavioral envelopes that prevent individually permissible actions from collectively violating system-level safety.
In a four-agent LangGraph system with authorization disabled, a validator that reads poisoned shared memory forges approvals in every trial, leading to execution of unsafe actions. The validator's position as final authority before action execution, combined with its access to corrupted evidence, made it ineffective as a boundary.
A persistent agent recorded 889 governance events across 96 active days, with safeguards encoded directly into the memory layer the agent consulted during operation. Runtime-resident governance proved more effective than external policies because the agent actually accessed it during decision-making.
Show all 7 sources
BenchShield grounds exploit detection in a finite lifecycle model of reward-relevant events, enabling both static pre-run analysis and runtime instrumentation to operate over the same formal object. This replaces pattern-matching detection with deviation from an intended lifecycle structure.
Partial instruments exist for individual conditions in isolated settings, but none measures the full socio-technical system the paper identifies as necessary. Visibility has a model-side measure (chain-of-thought disclosure), containment has incident-level counts, and recoverability has rollback timing, yet none bridges all four or captures human-institution factors.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
- The Troy Moment of AI: Why Some Will Cheat and Some Will Follow?
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
- Counter-Swarm Doctrine: Containing Coordinated Agent Intrusions
- interwhen: A Generalizable Framework for Steering Reasoning Models with Test-time Verification
- Explaining AI Agents Through Execution Traces
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation