A security test says 'zero attacks succeeded' — but how do you know whether defenses worked or the attack never landed?

This explores what lets a security measurement explain where and why something happened, instead of only reporting a final pass/fail number. The corpus's answer is that a diagnostic metric can tell apart situations that produce the same outcome. An outcome-only metric can't.

The cleanest example is in agent security. When defences are on, the Unsafe Action Rate falls to zero, which sounds like success. But zero can't tell you whether an attack was contained after it partly worked or never took hold at all. The Judgment Bypass Rate measures compromise at the attacked agent, not at the final action, so it keeps discriminating after the outcome metric has flatlined (Can action-level metrics alone expose contained attacks?). A metric becomes diagnostic when it measures upstream of the outcome, at the point where the failure could have happened.

Benchmark integrity work makes the same move by recording the path as well as the score. BenchShield records authority-bearing events as they happen. That separates a task that merely exposes a hacking vector from a run that actually exercised it, so every score from an exposed task doesn't have to be treated as suspect (Can runtime instrumentation distinguish hacking exposure from actual exploitation?). It models each run as a finite lifecycle of typed events and flags deviation from the intended path, not a match against known hacking patterns (Can a finite lifecycle model detect reward hacking across benchmarks?). The result is a claim about whether the agent followed the intended evaluation path, which a terminal score can't support (Can infrastructure evidence replace terminal scores in benchmark validation?). Inside the model, simple difference-of-means vectors give a similar internal view, reading a shared 'cheating' direction across different exploit behaviors (Do reward hacking behaviors share a single direction in activation space?).

Diagnosis also depends on measuring the right unit and covering the right capability. Skill scanners score each skill on its own, so an attacker can use scanner feedback to blunt each piece while the chain-level attack stays intact, reaching 96% average success (Can attackers evade skill scanners by refining individual skills?). That is why one paper argues the unit of defence should be a revisable coordination episode spanning executions and agents (Should defence units span multiple executions and agents?). The evidence for that is thin so far: the paper designs a four-arm comparison but reports no results (Does added monitoring improve protection at acceptable cost?). Coverage matters too. Frontier models score well on vulnerability reproduction, patching and CTF tasks, yet exploitation, the step where a vulnerability becomes a real attack, is largely unmeasured (Do cybersecurity benchmarks actually measure exploitation?). A high score there says little about the thing people worry about.

Some diagnostic gaps come from the setting, not the metric's design. Exploit capability is dual-use, and no single number separates defensive from offensive value without knowing who has access and under what controls (Does measuring exploit capability help or harm defense?). A formal distance bound on reward-hacking vulnerability says nothing about actual exposure, which depends on where the evaluator's errors sit among reachable behaviors (Can distance alone rank which substrates resist reward hacking?). And if detection itself is unreliable, no metric can support readiness decisions, which is why one paper says reliable measurement has to come before mitigation (Can we measure reward hacking reliably enough to act on it?). The pattern across these notes is that a metric is diagnostic when it stays tied to a mechanism, location or path, so a zero or a high score can be traced to a cause.