When an AI attack 'succeeds' or fails, is that the model itself, or a hidden filter in front of it?

This explores whether an attack-success number can be measuring a filter instead of the model. The corpus says it can, and the field mostly doesn't say which numbers came from behind one. A vault-wide audit found that attack success and defense gain percentages are usually reported without saying whether they were measured behind server-side filters, so a reader can't tell if a result reflects the model's own behavior or a filtered outcome Which attack and defense numbers came from filtered backends?.

The ambiguity runs both ways. A low attack success rate could mean the model resisted, or that a filter dropped the request before the model saw it. A large defense gain could come from the new defense, or from a filter that was already there. That reading is my inference, but it follows directly from the missing labels. The routing note adds another layer. The step that decides which model handles a request can itself be manipulated, and its provenance often goes unverified. That can send requests to weaker models or make safety measures operate on the wrong identity Can attackers manipulate which model handles a request?. So even the name of the model attached to a number may not be what was tested.

A good-looking number behind a filter also doesn't show the system is safe. A filter judges one output at one moment, but an agent's risk spreads across memory, retrieved content, tool calls and what it can reach in its environment Can a model-level filter truly contain an agent with environment access?. A low single-turn success rate therefore says little about the agent as a whole. ColluSkill shows the opposite failure. It reached 96% average success against six skill scanners by using scanner feedback to blunt each skill while the chain-level attack stayed intact Can attackers evade skill scanners by refining individual skills?. That figure describes an attacker tuned against one particular defense stack. It doesn't describe the model underneath.

The corpus points to one fix. BenchShield lets operators back a benchmark claim with recorded infrastructure evidence instead of a terminal score, turning a single number into a checkable claim about whether the agent followed the intended evaluation path Can infrastructure evidence replace terminal scores in benchmark validation?. That note is about benchmark validity, not attack numbers. Applying the same idea to filter status is my extension, not something it claims. The practical habit is to ask what stood between the attacker and the model before trusting the percentage.