How do agent risks accumulate across long stateful workflows?
Current safety benchmarks test isolated tasks, but real agents operate in persistent environments where early decisions ripple forward. Does cumulative risk from evolving state differ fundamentally from per-action risk?
The OpenART abstract makes a claim about where agent risk lives. Agents "operate in persistent environments where early state changes can influence decisions far into the future," and their behavior "is mediated through shared state that is repeatedly read, modified, and reused across long-horizon workflows." From that the paper concludes that "safety therefore depends not only on individual actions, but also on how agents respond as environments evolve over time." The unit of safety analysis moves from the single action to the sequence of actions taken against changing state.
The paper's diagnosis of current practice follows from this. Existing agent safety benchmarks "primarily evaluate short, static tasks, making it difficult to study cumulative risks in evolving environments," and benchmark-specific interfaces "hinder direct comparison across agent runtimes." Two separate gaps are named: the tasks are too short and static to show risk that builds up, and the interfaces are too particular to let results travel between agents. OpenART's response is scale on both fronts. It builds over 10K validated stateful scenarios across 50 domains, drawn from more than 500K Tools, MCPs, and Skills, with tasks needing a median of 97 tool calls. Target adapters project these onto 15 deployed agents, 5 foundation models, and 8 attack vectors, giving 75 agent–model configurations under one evaluation.
This sits close to the argument in Does a single benchmark score actually predict agent readiness?, which holds that no single benchmark captures readiness. OpenART adds a temporal axis to that list: a benchmark can score every action correctly and still miss what happens when an early state change carries forward. It also contrasts with What failure modes emerge when agents operate without direct oversight?, where twenty researchers found failures by hand over two weeks; OpenART aims at the same kind of adversarial probing but as a generated, repeatable arena. And where What blocks scaling from language models to autonomous agents? treats environments as something to scale for training, OpenART scales them for evaluation, with domain count and task length as the visible axes.
The excerpt is only the abstract, so much is left open. It does not say how environments "evolve," how scenarios were validated, what the 8 attack vectors are, or what any agent or model scored. It reports no evidence that cumulative risk is larger than per-action risk; that is the paper's premise, not something the excerpt demonstrates. The defensible reading is that OpenART is a proposal to measure state-dependent risk at scale, and its argument for why such measurement is needed is stated but not yet tested in what we have.
Inquiring lines that read this note 1
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Why do locally safe actions create system-level safety gaps?Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does a single benchmark score actually predict agent readiness?
Single-axis benchmarks rank models by one capability—like task success—but ignore privacy, duration, operating mode, and ecosystem fit. Can one number really capture what matters for deployment?
extends the multi-axis evaluation argument by adding long-horizon, state-dependent behavior as a further axis
-
What failure modes emerge when agents operate without direct oversight?
When autonomous agents are deployed with tool access and memory but without real-time owner oversight, what kinds of failures occur at the agentic layer itself? Understanding these patterns matters for safe deployment.
manual two-week red teaming versus a generated, repeatable arena for the same kind of probing
-
What blocks scaling from language models to autonomous agents?
If large language models excel at next-token prediction, why do they struggle with long-horizon goal-oriented tasks? This explores whether the bottleneck is model capacity or the environments used to train them.
scales environments for training; OpenART scales them for safety evaluation
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution
- Agents of Chaos
- Why Do Multi-agent LLM Systems Fail?
- FLOWSTEER: Prompt-Only Workflow Steering Exposes Planning-Time Vulnerabilities in Multi-Agent LLM Systems
- Emergent Collusion in Long-Horizon LLM Agent Interaction
- Unaccountable Delegation, Fading Skills: Mapping the Risks of Workplace AI Agents
- AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
- Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
Original note title
agent safety depends on how agents respond as shared state evolves over long workflows — short static benchmarks make cumulative risk hard to study