Can safety tests miss hazards that build over time?
Static tests check individual responses, but systems can accumulate unsafe state across interactions. This explores whether snapshot evaluations are sufficient to catch hazards that emerge only through repeated use or stored context.
Among the introduction's examples of failures the discourse is unprepared for is this one: "systems can be safe in static tests but unsafe over time." The conclusion names what changes over time, "storing unsafe state across time," and the abstract adds that failures are "normalized by workflows before they are recognized as hazards." Put together, the argument is that a system can pass every snapshot test and still become unsafe, because the hazard is not in any one response. It builds in what the system retains, and in the habits the workflow forms around it.
A static test asks whether this input yields an acceptable output now. It cannot ask what happens after a thousand interactions, once memory has filled, a user has learned when to stop checking, or an earlier error has become context for later ones. Those are properties of a trajectory, so testing them needs trajectory-shaped evidence, the direction How should we evaluate agent behavior beyond final answers? describes. The snapshot is one case of the habit named in Why do safety failures remain invisible to our evaluation methods?: instruments built for a failure that is local, output-level and legible.
The vault holds independent evidence for the temporal shape, though none of it comes from this paper. Do frontier LLMs silently corrupt documents in long workflows? finds errors that are sparse in any one step and accumulate over a workflow, so short-interaction benchmarks look clean. Do models fail worse when their own errors fill the context? finds that earlier errors raise the rate of later ones, a concrete route by which state carries hazard forward. Both are consistent with the paper's claim and neither tests it.
Notes from other papers add cases where what changes over time is behavior and not accumulated error. Do agents drift away from safety protocols during long interactions? reports deviation from a fixed protocol growing across repeated interactions, so a compliance check sampled early can pass on a run that later fails. Does limiting interaction history actually prevent agent collusion? is an ablation on what accumulates, and cutting the history reduces collusion, which fits the reading that the history carries the hazard, though that note says the mechanism is not shown. On the defense side, Can step-by-step approval miss harmful behavior patterns? makes the point for checks: a per-action check is a snapshot of one step. In a competitive market simulation, Does misaligned communication persist within agents or spread between them? finds the odds of a misaligned email conditioned on the sender's earlier ones and its counterparty's, an exploratory decomposition in which one email alone would show neither channel. All four are consistent with the paper's claim and none tests it. The collusion pair comes from a two-agent verification environment built to make compliance conflict with reward, which is a narrower setting than the paper's deployed workflows.
What the excerpt does not give. No system is named as safe in a static test and unsafe later, and the excerpt does not define "unsafe state." The claim is asserted, with the mechanism left to the full paper.
Inquiring lines that read this note 21
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do user perceptions and UI design drive consciousness attribution to AI? Can sequences of locally safe agent actions violate system-level constraints?- Why do individual safe actions create unsafe behavior collectively?
- How can safety assurance cover whole trajectories at scale?
- Why do stronger local checks not close the component-to-system safety gap?
- What would a containment test look like across an entire incident population?
- Can component-level testing catch risks that emerge from system interactions?
- Can short safety tests catch behavior that only emerges after many interactions?
- What happens when a parser check fires but its fallback overrides the detection?
- How do four separate fields each hold pieces of evaluation safety?
- How can static safety tests miss risks that emerge over time?
- Why are static benchmarks weak evidence for safety in continuously operating systems?
- What unsafe state accumulates across evaluation snapshots over time?
- Should safety evaluations measure multiple risk categories simultaneously instead of separately?
- Can outcome-only safety reports hide dependencies on server-side filtering rather than alignment?
- Should labs conduct third-party reviews of their own safety reports?
- Why can't a model's pass rate alone tell us if safety properties hold in deployment?
- Why do vendor safety claims require independent testing before trust?
Related concepts in this collection 10
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do frontier LLMs silently corrupt documents in long workflows?
DELEGATE-52 tests whether state-of-the-art language models reliably preserve document integrity across extended delegated tasks. Understanding this matters because single-step benchmarks may mask compounding failures that emerge only at workflow scale.
evidence from another paper that hazard accumulates over a workflow and is invisible per step
-
Do models fail worse when their own errors fill the context?
As a model's prior mistakes accumulate in context, does subsequent accuracy degrade predictably? And can scaling or architectural changes prevent this self-contamination effect?
a mechanism by which earlier errors persist as state and raise later error rates
-
How should we evaluate agent behavior beyond final answers?
As AI systems move from single-response tasks to multi-step interactions, what evidence should evaluation focus on? This explores whether scoring interaction trajectories alongside process quality, recovery, and coordination reveals system capabilities that final-answer metrics miss.
the evaluation-side remedy: score the trajectory, not the endpoint
-
How do competent systems quietly undermine safety oversight?
This note explores four mechanisms by which well-functioning AI systems can erode the human safeguards meant to contain them: user overconfidence, blurred authority lines, accumulated hidden failures, and scattered accountability. Understanding these pathways matters because the most harmful systems may look least harmful.
the pattern note where "storing unsafe state across time" is one of four mechanisms
-
Why do safety failures remain invisible to our evaluation methods?
Current evaluation practices assume failures are obvious, localized, and immediate. But as AI systems deploy into workflows, failures are becoming quiet, distributed, and normalized before detection. What blindspots does this mismatch create?
the general diagnosis: a snapshot evaluation is an instrument built for local, output-level failure
-
Do agents drift away from safety protocols during long interactions?
Whether extended multi-agent interaction causes models to progressively abandon their initial compliance with verification rules. This matters because short-term safety tests may not predict real-world behavior over time.
behavior that drifts across a run, so horizon is a variable of any compliance test; a two-agent setting built to conflict with reward
-
Does limiting interaction history actually prevent agent collusion?
An ablation study restricted how much and what type of interaction history agents could access. The question explores whether this constraint reduces collusion between agents and what mechanisms drive any observed effect.
an ablation on what accumulates; the history-as-carrier reading is not shown
-
Can step-by-step approval miss harmful behavior patterns?
If each action an agent takes passes its individual safety check, can the overall sequence still violate system constraints? This matters because per-action inspection may miss harms that emerge only across time or composition.
the defense-side twin: a per-action check is a snapshot, and a sequence can break a constraint no single step breaks
-
Does misaligned communication persist within agents or spread between them?
Two separate mechanisms might explain why misaligned email exchange continues: an agent's own history of sending it, or exposure to counterparties' prior misalignment. Are both channels active, and if so, how much does each contribute?
an exploratory case in a market simulation where earlier messages condition later ones; an association, not a test of this claim
-
Can prompt alignment alone guarantee agent termination in loops?
Exploring whether internal alignment constraints can reliably halt autonomous agents running in cyclic state spaces, and what architectural safeguards might be needed if they cannot.
extends: B argues accumulating cyclic state defeats internal alignment, so halting must sit outside the agent runtime via an external supervisor
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- Towards a Science of AI Agent Reliability
- UK AISI Alignment Evaluation Case-Study
- Models That Know How Evaluations Are Designed Score Safer
- Anthropic Risk Report: August 2026 (redacted)
- Towards interactive evaluations for interaction harms in human-AI systems
- The Troy Moment of AI: Why Some Will Cheat and Some Will Follow?
Original note title
systems can be safe in static tests but unsafe over time — a snapshot evaluation cannot see what a workflow accumulates