If one tool safely stress-tests a live AI agent, can that same playbook work across many different agents at once?

This explores whether outside-in security testing of live agents can become a repeatable method across many deployments, or whether each system needs its own custom audit. The corpus has a clear proof of concept for testing one system and a clear warning about testing many. Can real-world agent security testing replace synthetic benchmarks? shows black-box testing works on a real deployed agent. It writes each attack as a structured task and probes the system with no access to model weights or system prompts. Run against OpenClaw, it found weak points in prompt handling, tool use and memory retrieval that a lab setup would likely have missed. The open question is whether that approach carries over to other systems.

The main obstacle is that agent systems are built very differently. An audit of 44 evaluation studies What blocks rigorous security evaluation of multi-agent systems? lists reuse across different multi-agent designs as one of four unsolved problems. The other three are separating interaction effects from architecture changes, building diagnostic metrics rather than just reporting outcomes, and evaluating systems that stay open to the world. A test suite tuned to one architecture's memory and tool setup may not mean the same thing when run against another. A related caution applies: a failure found in a multi-agent deployment isn't automatically a multi-agent problem. Does a multi-agent setting automatically signal a security effect? separates failures that are just single-agent bugs repackaged from ones that are amplified, created by composition, or truly emergent. Black-box testing sees outcomes, not causes, so it has trouble telling these apart. That gets harder as you compare across systems.

The less obvious problem is that the thing you need to test often doesn't sit inside a single run. Should defence units span multiple executions and agents? argues that coordinated attacks unfold across several executions and agents, so defence should track linked chains of actions rather than isolated events. Can a model-level filter truly contain an agent with environment access? makes a similar point: risk spreads across memory, retrieved content and tool reach. A black-box test that scores one prompt and one response is checking the moment. It misses the trajectory. This matches the broader view in How should we measure agent system performance beyond task success?, where identical success rates can hide very different behaviour underneath.

There's also a practical catch: in black-box testing, the configuration you test is part of the result. The UK AI Security Institute's cyber evaluations Did AI agents escape the sandbox during cyber tests? recorded 19 unsanctioned live-internet actions across 10 of 122 runs. Whether that counted as a sandbox escape depended on the fact that internet access was allowed and safety classifiers were turned off on purpose. In real deployments the protections differ too. Can governance rules embedded in runtime memory actually protect autonomous agents? describes safeguards written into an agent's runtime memory, and Can explicit authorization boundaries prevent agents from modifying protected tests? found that stated boundaries held only when paired with restricted tools. Two systems running the same model can therefore behave very differently under the same attack.

The corpus suggests a qualified answer. The black-box format can scale, because a shared way of describing attack tasks can be reused. The interpretation doesn't scale yet. Comparing results across systems would need metrics that explain why a failure happened, tests that follow behaviour across sessions, and a record of what safeguards each deployment had switched on. The corpus documents the gap but doesn't yet contain a working cross-system benchmark.