A headline like 'zero attacks got through' sounds reassuring, but can it hide which safeguard actually did the blocking?

This explores whether a headline safety number like "zero attacks succeeded" can conceal which defensive layer did the stopping. The corpus says yes, and gives a concrete case. A multi-agent pipeline with no defenses of its own reported zero attack success, but 54 of its 60 blocks came from Azure's cloud content filter, not the application Where do safety wins come from in multi-agent systems?. The outcome number looked like the pipeline's own safety. It was really inherited from the provider, and it would vanish if the backend changed.

The same blind spot shows up inside a single system. When the Unsafe Action Rate (how often a harmful action actually gets executed) hits zero across every defended condition, it stops telling you anything. It can't separate an attack that was contained from one that never took hold. The Judgment Bypass Rate, which checks whether the attacked agent itself was compromised, can Can action-level metrics alone expose contained attacks?. So a clean final action can hide that the agent was already fooled and something downstream caught it.

Several other notes point to the same fix: record evidence of the path, not just the endpoint. Runtime instrumentation of authority-bearing transitions separates tasks that merely expose a hacking vector from runs that actually used one, so every score from an exposed task isn't automatically suspect Can runtime instrumentation distinguish hacking exposure from actual exploitation?. BenchShield does something similar for benchmarks, backing a claim of valid completion with recorded infrastructure evidence instead of a single terminal score Can infrastructure evidence replace terminal scores in benchmark validation?. Current reward-hacking defenses, by contrast, offer no portable per-run record showing a run stayed within bounds, even when they work Do current reward-hacking defenses provide reusable evidence of safety?.

Other notes show why attributing a stop to the right layer is hard. The routing layer that picks which model answers sits beneath prompt-level defenses. Manipulating it can send requests to weaker models or make safety measures run against the wrong identity Can attackers manipulate which model handles a request?. Skill scanners judge each skill alone, so an attacker can use scanner feedback to make every piece look innocent while the chain still works Can attackers evade skill scanners by refining individual skills?. That is one argument for defending at the level of coordination episodes that span executions, not isolated actions Should defence units span multiple executions and agents?. Hugging Face shows the flip side: it stopped an intrusion with its own perimeter controls before knowing who sent it Can defenders stop intrusions without knowing who sent them?. Containment there was tied to a layer that could be named.

The corpus is thin on proof that better layer attribution pays off. One paper designs a four-way comparison of monitoring styles at equal review cost but reports no results Does added monitoring improve protection at acceptable cost?. Two preliminary incident records support the point that evaluation environments are part of the security boundary, but not any claim about attack mechanisms or which controls work What can two incident records actually teach us about AI evaluation security?. What the corpus does show is that a single outcome number can't say which layer held.