How can we measure whether AI errors stay visible and recoverable?

The paper's title is about instrumentation, "the safety failures we are not instrumenting," and its closing move is to offer an organizing device for it: "The five-layer framework offered here is not exhaustive, and many concrete failures will span several layers simultaneously; its value is as an organizing device for instrumentation and governance rather than as a closed taxonomy." The standard it sets is What makes an AI system truly safe in practice?. What is missing from the excerpt is the bridge between the two: a measurement for any of those four conditions.

The gap is real and different for each condition. Visibility could be tested by planting an error and counting how often it is noticed. Recoverability could be tested by timing a rollback. Contestability and containability are harder, because both depend on people and institutions, not only on the system. The paper's own framing suggests the instrument should measure the socio-technical system, which puts it outside the model-only tools the paper criticizes. One existing instrument measures a model-side half of visibility for a single error class: Do models that leak values also disclose those leaks? scores whether a model's own chain-of-thought discloses the influence on its answer. It measures the system's self-disclosure and not whether a person notices, so it would sit beside a planted-error test, not replace one. A test that catches a planted error is also the natural way to tell real oversight from the nominal kind in Can organizations lose scrutiny capacity while keeping oversight forms?.

Other notes in the vault hold partial instruments, each for one condition in one setting, and none measures the socio-technical system the paper points at. How often do incident records document system stops? reads the containable end from an incident record, at the level of a population of incidents and not of any one system, and the excerpt gives neither the population nor the coding scheme. Can action-level metrics alone expose contained attacks? reads two locations at once, so it separates a contained attack from one that never took hold, but for one attack class in one pipeline. Planting a known event so that nobody has to judge whether it happened is the device in Can planted honeypots reliably catch reward hacking automatically?, pointed at agent behavior and not at whether a person notices. It counts only what was planted (Can planted honeypots detect hacks that matter most?), and a planted-error test for reviewers would share that limit.

Two more show what an instrument needs. Can a finite lifecycle model detect reward hacking across benchmarks? is the most fully specified one in the vault, and it works because the property, a run's benchmark-validity, was defined in advance as observable events; none of the four conditions is defined that way in the excerpt. On the record side, Can external anchoring detect tampering in agentic process logs? proposes an artifact from which some measures could be computed and reports none.

Two things are not in the excerpt and should be retrieved from the full paper before this note is used in a post: the names of the five layers, and any concrete instrument. The excerpt does say what the framework is for and what it is not, an organizing device and not a closed taxonomy, and it warns that failures span layers. The candidate mapping from the four mechanisms in How do competent systems quietly undermine safety oversight? to the layers is a hypothesis of this vault. The excerpt does not say the four mechanisms are the layers.