When an AI system scores 'zero attacks succeeded,' is the model actually safe — or is a filter in front doing the work?

This explores how a reported "zero attack success" can come from a server-side filter in front of the model, not from the model or defense being tested. The short answer the corpus gives is that filters hide by omission. Attack success and defense-gain percentages across the vault are mostly reported without saying whether they were measured behind a filter (Which attack and defense numbers came from filtered backends?). A 0% then reads as "the model resisted," but nothing in the number tells you whether the model, the defense, or the filter did the work.

The same gap shows up when papers bundle several protections and report one zero. One pipeline reports zero protected-test modifications for a package of clear authorization rules plus restricted tools. No ablation separates "the agent couldn't cross the line" from "the agent chose not to" (Do authorization rules or restricted tools prevent test modifications?). The same pipeline also shows a 100% Judgment Bypass Rate alongside a 0% Unsafe Action Rate. So the model's judgment was bypassed every time and nothing unsafe happened, which means something downstream produced the zero. Another paper reports zero unsafe actions with authorization enabled, but the excerpt gives only two phrases about how it works and never says whether the attacks were positioned to reach those components (How does the authorization layer stay outside the poisoned path?).

A filter's zero is also weaker than it looks. A filter judges one output at one moment, while an agent's risk spreads across memory, retrieved content, tool calls and environmental reach (Can a model-level filter truly contain an agent with environment access?). Zero on filtered outputs therefore says little about containment. The zero also depends on the attacker not adapting. When attackers get scanner feedback, they reach 96% average success against six scanners, because each skill is scored alone while the harmful chain stays intact (Can attackers evade skill scanners by refining individual skills?). Any rule that separates safe from unsafe becomes a roadmap once the attacker knows it (Can honeytokens fool attackers who know the trusted policy?). And if the layer that routes requests can be manipulated, safety measures may end up running on the wrong model identity (Can attackers manipulate which model handles a request?).

None of this makes filters illegitimate. Hugging Face stopped an intruding agent using only its own perimeter controls, without knowing who sent it (Can defenders stop intrusions without knowing who sent them?). A zero is a claim about conditions, and even the theoretical zero-error result for repeated probes holds only when responses are independent and their distributions are known or learnable (Can repeated quiet probes separate decoys from genuine objects?). The corpus doesn't pin any specific zero on a filter. What it shows is that the missing label makes the question unanswerable. When you see 0%, ask what sat in front of the model, whether the attack ever reached it, and whether the attacker could see or adapt to the filter.