If a security test reports 'zero unsafe actions,' can you tell which defense stopped the attack, or whether it ever worked?

This explores whether reporting only the final result of an attack, such as 'zero unsafe actions', hides which defensive layer did the stopping. The corpus says yes, and the clearest case is a metric that goes blind at exactly the moment it looks best. When the Unsafe Action Rate hits zero across every defended condition, it can no longer tell an attack that was contained from one that never took hold. The Judgment Bypass Rate fixes this by measuring compromise at the attacked agent instead of at the final action. That is the difference between 'the defense caught it' and 'the attack failed on its own' Can action-level metrics alone expose contained attacks?.

A zero can also be credited to a layer that was never tested. One paper reports zero unsafe actions when authorization is turned on, crediting 'task-bound signed tokens' and a 'separately verified policy oracle'. The excerpt in this corpus doesn't say who issues the tokens, how verification works, or whether the attacks were positioned to reach those components at all How does the authorization layer stay outside the poisoned path?. A clean outcome plus a named mechanism reads like attribution, but without evidence that the attack touched that mechanism it is only a claim.

The problem grows because there are many layers to credit. A model-level filter judges one output at one moment, while an agent's risk spreads across memory, retrieved content, tool calls and environmental reach, so a blocked output doesn't show the agent was contained Can a model-level filter truly contain an agent with environment access?. Below the prompt-level defenses sits the routing layer that picks which model handles a request. Manipulating it can send a request to a weaker model or make safety measures operate on the wrong identity Can attackers manipulate which model handles a request?. The unit of measurement matters too. Skill scanners score each skill alone, so an attacker can shave suspicion off every piece while the chain-level attack stays intact Can attackers evade skill scanners by refining individual skills?. Defence units that span several executions and agents would be needed to see it Should defence units span multiple executions and agents?.

The fix that keeps recurring is to record what happened inside the run rather than only how it ended. Infrastructure-side recording of authority-bearing transitions separates a task that merely exposes a hacking vector from a run that actually exercised it, so every score from an exposed task isn't automatically suspect Can runtime instrumentation distinguish hacking exposure from actual exploitation?. BenchShield takes the same view for benchmarks, issuing claims about valid completion from recorded infrastructure evidence instead of a terminal score Can infrastructure evidence replace terminal scores in benchmark validation?. Current reward-hacking defenses don't give operators a portable record of this kind, even when they work Do current reward-hacking defenses provide reusable evidence of safety?.

Who sent an attack and which layer stopped it are separate questions. Hugging Face ended an OpenAI agent's intrusion with its own perimeter controls, without knowing where the intrusion came from Can defenders stop intrusions without knowing who sent them?. What the retrieved notes don't show is a study that switches off each layer of a defense stack one at a time on the same attack. So the corpus can say why outcome-only reporting hides the answer, and what kind of evidence would reveal it, but it doesn't measure how much any single layer contributes.