A safety test reported zero successful attacks, but the cloud provider's filter did the blocking, not the system itself.

This explores how a headline safety number, like "zero attacks succeeded," can conceal that a filter outside the system being tested did the blocking, and why that matters for anyone trusting the result. The corpus has a concrete case. A multi-agent pipeline with no defenses of its own was tested across several backends and reported zero attack success. But 54 of the 60 blocks came from Azure's cloud filter, not from the application Where do safety wins come from in multi-agent systems?. An outcome-only report can't tell "the system is robust" from "the provider is protecting us." The dependence stays invisible until someone swaps the backend and the protection disappears.

The problem looks widespread. Across the vault's published attack and defense numbers, most are reported without saying whether they were measured behind a server-side filter Which attack and defense numbers came from filtered backends?. So a reader can't tell whether a figure reflects the model's real behavior or a filtered version of it. A defense that appears to add 30 points of protection might just be riding on a filter that was already there.

The same blind spot shows up in other forms. A verdict that matches ground truth can hide agents that skipped required verification steps, so a correct outcome doesn't prove the protocol was followed Can a correct outcome hide protocol violations in multi-agent systems?. Bundled safety regimes have the same trouble. One paper combines clear authorization rules with restricted tools and reports zero protected-test modifications. Without single-factor ablations, you can't tell whether the result came from crossings that were unavailable or crossings the agent chose not to make. The paper's own data elsewhere, a 100% Judgment Bypass Rate alongside 0% Unsafe Action Rate, shows why that difference matters Do authorization rules or restricted tools prevent test modifications?. More generally, safety failures often hide in evaluation habits rather than technical obscurity: our instruments expect shocking, localized failures, and the real ones are plausible and distributed Why do safety failures remain invisible to our evaluation methods?.

The filter itself also has limits worth knowing. A model-level filter judges one output at one moment, while an agent's risk spreads across memory, retrieved content, tool calls, and reach into its environment. Blocking a moment of behavior is not the same as containing an agent Can a model-level filter truly contain an agent with environment access?. So a clean score behind a filter can overstate safety in two ways. The credit goes to the wrong layer, and even the right layer may cover less than it seems to.

The corpus points to a few fixes. Label whether each result was measured behind a filter, or better, report attack success with and without the filter. Ground claims in recorded evidence of how a task was completed, as BenchShield does with infrastructure records instead of a terminal score Can infrastructure evidence replace terminal scores in benchmark validation?. And state the scope of any statistical guarantee, since pointwise, average-case and worst-case claims aren't interchangeable What scope should a validator's statistical guarantee actually state?. The corpus doesn't yet show whether adding more monitoring pays off. One paper designs that comparison but reports no results Does added monitoring improve protection at acceptable cost?.