If an AI agent might slip its leash, how do you test whether it stays contained across many incidents at once?

This explores what it would take to test whether containment (keeping a capable AI agent inside its intended boundaries) holds across a whole population of incidents, not one incident at a time. The corpus has a concrete design on paper but no results. One paper proposes a four-arm comparison: watch isolated actions, rolling windows of activity, known groups of related actions, and episodes discovered as events unfold, all at the same review cost and false-alert workload (Does added monitoring improve protection at acceptable cost?). The question that test asks is which way of carving up the incident population catches more for the same effort. It does not ask whether monitoring works in general.

The grouping matters because of what the unit of defence should be. One line of work argues it should be a 'coordination episode': a set of actions linked by observed transfers, task authority, and response history, with membership revised as evidence comes in (Should defence units span multiple executions and agents?). Reviewing each execution alone loses context that spans several executions. A separate result shows how this can fail in practice. The same security instructions cut payload exposure by 40–75% for single agents and by zero for multi-agent systems, because the agent doing the risky thing didn't hold the security context (Why does prompt hardening work for single agents but not multi-agent systems?). A population-wide test therefore has to include multi-agent incidents. A test built from single-agent cases would report protection that disappears where the coordination happens.

The test also has to run over time and record its own conditions. Systems can pass every snapshot test and still become unsafe, because hazards build up in stored state and routine workflows rather than in any single response (Can safety tests miss hazards that build over time?). So it should score trajectories, not isolated outputs. A benchmark-style score also hides containment by construction: two labs can report the same number under very different containment levels (What do benchmark scores actually reveal about model containment?). Each result would need to carry its containment setup with it. That setup has to cover the evaluation environment (Is your evaluation environment actually part of the threat model?) and the response workflow around the agent (Should response workflows be inside the security boundary?), because both count as part of the boundary once an agent has tools and credentials.

Which incidents get sampled is the next design choice. Persona-simulation research finds that optimizing for support coverage beats matching the statistical density of typical cases, because coverage reaches rare but consequential configurations that ordinary sampling misses (Should persona simulation prioritize coverage over statistical matching?). The same logic suggests a containment test should deliberately include unusual incident shapes and not just mirror the common ones. The adversary side gives a second axis to measure, which is time. If probing triggers containment, an intruder has to gather enough observations before being removed, and finite-sample bounds put a price on that (How fast must a coalition gather observations before containment?). In idealized settings, enough quiet probes will separate decoys from genuine objects (Can repeated quiet probes separate decoys from genuine objects?). A test could therefore ask how many quiet observations get through before containment fires.

The corpus is thin on the baseline such a test needs. The two incident records analysed so far support the lesson that evaluation environments are inside the security boundary, but they don't establish common attack sequences, recurrence rates, or which controls work (What can two incident records actually teach us about AI evaluation security?). Capability measurement and containment research also still sit in separate literatures, joined only by a first grouping into five vulnerability classes (How do we contain capable agents during evaluation?). A population-level containment test would first have to build the incident population it is meant to test against.