How can we measure whether AI errors stay visible and recoverable?
The paper proposes four conditions for safer AI systems—visibility, contestability, containability, and recoverability—but lacks concrete measures for any of them. What would it take to instrument each condition across the socio-technical system?
The paper's title is about instrumentation, "the safety failures we are not instrumenting," and its closing move is to offer an organizing device for it: "The five-layer framework offered here is not exhaustive, and many concrete failures will span several layers simultaneously; its value is as an organizing device for instrumentation and governance rather than as a closed taxonomy." The standard it sets is What makes an AI system truly safe in practice?. What is missing from the excerpt is the bridge between the two: a measurement for any of those four conditions.
The gap is real and different for each condition. Visibility could be tested by planting an error and counting how often it is noticed. Recoverability could be tested by timing a rollback. Contestability and containability are harder, because both depend on people and institutions, not only on the system. The paper's own framing suggests the instrument should measure the socio-technical system, which puts it outside the model-only tools the paper criticizes. One existing instrument measures a model-side half of visibility for a single error class: Do models that leak values also disclose those leaks? scores whether a model's own chain-of-thought discloses the influence on its answer. It measures the system's self-disclosure and not whether a person notices, so it would sit beside a planted-error test, not replace one. A test that catches a planted error is also the natural way to tell real oversight from the nominal kind in Can organizations lose scrutiny capacity while keeping oversight forms?.
Other notes in the vault hold partial instruments, each for one condition in one setting, and none measures the socio-technical system the paper points at. How often do incident records document system stops? reads the containable end from an incident record, at the level of a population of incidents and not of any one system, and the excerpt gives neither the population nor the coding scheme. Can action-level metrics alone expose contained attacks? reads two locations at once, so it separates a contained attack from one that never took hold, but for one attack class in one pipeline. Planting a known event so that nobody has to judge whether it happened is the device in Can planted honeypots reliably catch reward hacking automatically?, pointed at agent behavior and not at whether a person notices. It counts only what was planted (Can planted honeypots detect hacks that matter most?), and a planted-error test for reviewers would share that limit.
Two more show what an instrument needs. Can a finite lifecycle model detect reward hacking across benchmarks? is the most fully specified one in the vault, and it works because the property, a run's benchmark-validity, was defined in advance as observable events; none of the four conditions is defined that way in the excerpt. On the record side, Can external anchoring detect tampering in agentic process logs? proposes an artifact from which some measures could be computed and reports none.
Two things are not in the excerpt and should be retrieved from the full paper before this note is used in a post: the names of the five layers, and any concrete instrument. The excerpt does say what the framework is for and what it is not, an organizing device and not a closed taxonomy, and it warns that failures span layers. The candidate mapping from the four mechanisms in How do competent systems quietly undermine safety oversight? to the layers is a hypothesis of this vault. The excerpt does not say the four mechanisms are the layers.
Inquiring lines that read this note 150
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can AI systems achieve sustainable self-improvement without external human guidance? Can sequences of locally safe agent actions violate system-level constraints?- Can export control tools stop deployed AI models without legal redesign?
- How do you stop an AI system once it is already deployed?
- Does visibility and contestability of errors replace prevention as the safety goal?
- Why do stronger local checks not close the component-to-system safety gap?
- What authority should exist to stop an AI system once deployed?
- How were 'stop' mechanisms defined in the incident coding scheme?
- What does tight coupling mean in normal accident theory for AI?
- What counts as a successful stop or intervention on a deployed AI system?
- How often do deployed AI systems actually get stopped when they cause harm?
- What distinguishes containment and recovery from prevention as governance goals?
- How much capability do availability constraints remove on legitimate safe tasks?
- What tensions arise between user autonomy and platform safety in AI design?
- What containment methods prevent AI model attacks on out-of-scope third parties?
- How would hardware verification technology enable international agreements on AI safety?
- How can deployed AI systems be stopped once they are already in motion?
- Can standards enforcement prevent any single nation from accelerating unsafe AI research?
- What concrete baseline safeguards should global frontier AI standards actually require?
- Who has authority to halt a deployed AI system causing harm?
- Can error visibility alone improve AI system safety without containment?
- Does slowing AI development reduce risk or just delay it?
- Why is catching an AI red-handed treated as a win condition?
- What would mandatory external validation of AI safety look like?
- Do AI labs have insurance against catastrophic failure scenarios?
- Can stopping one AI security breach prove humans will retain control later?
- Does the substrate of a system change which dangerous AI behaviors become possible?
- What interface structure makes human control of AI agents most protective?
- Can AI development races be made safer by slowing down capability advancement?
- Does undo availability meaningfully increase psychological safety with AI systems?
- Can external process logs make AI errors verifiable and harder to hide?
- What does recovery look like as a formal part of AI design?
- How does automation obscure failure modes in ways that make detection harder?
- What distinguishes an error bound from a forecast of system behavior?
- Which evaluation habits keep safety-critical failures hidden in AI systems?
- How do workflows normalize and hide errors before they become visible hazards?
- What would it take to measure whether system errors stay visible and contestable?
- What does it mean for errors to remain visible, contestable, and recoverable?
- What makes a model's errors visible and contestable to users?
- How can a single instrument measure errors across multiple system layers?
- Why can't individual companies detect all emerging patterns in AI failures?
- Why do firms delay disclosing reliance on AI after errors surface?
- How does capability divergence from reliability affect AI deployment timelines?
- When should a company be responsible for an AI system's errors?
- How do users recover control when an AI system makes a mistake?
- What happens when AI systems fail to communicate their limitations to users?
- Can automatic compensation actively hide the failures it should reveal?
- Why do safety-critical failures stay hidden despite visible monitoring systems?
- Do technology experts show different error-detection patterns than general users?
- Why does real-time correction matter more than after-the-fact error catching?
- Why do probabilistic AI signals drift and become harder to monitor?
- Does contingency—like AI failure—actually reset the cycle of power displacement?
- Should low-confidence AI errors be framed as learning opportunities instead?
- What does invisible scaffolding mean for AI system design?
- Why do chat interfaces prevent users from actually checking AI outputs for errors?
- Can a single default apology strategy work across all AI failure modes?
- What counts as full capability recovery versus partial restoration?
- Can people reliably recognize when an AI is uncertain versus confident?
- Who can actually observe and challenge errors in multi-agent AI workflows?
- What does error recovery look like across different agent architectures?
- Does adding capability without improving detection reduce overall system reliability?
- How do OpenAI and Anthropic differ in categorizing agentic failure modes?
- Why do users regret AI agent actions based on visibility rather than output quality?
- Why is error rate alone misleading without strong contestability conditions?
- How should system safety aggregate when monitoring channels are unequal?
- Why do evaluation habits hide safety-critical challenges from view?
- How do response-centered evaluation assumptions hide safety-critical failure modes?
- What unsafe state accumulates across evaluation snapshots over time?
- Can deployed AI safety results hide either filters or unsafe model behavior?
- Would fragmented national AI standards make comparing safety evidence harder across labs?
- Why does AI industry culture prioritize speed over safety validation?
- Have AI safety evaluations adequately tested open-weight model capabilities?
- Do public benchmarks and leaderboards increase or decrease AI safety risks?
- How should AI safety violations be measured against probability and severity?
- What incident data validates GitLab's three-tier risk classification in production?
- Why do safety metrics miss silently dropped clinical obligations?
- What architectural changes make violations unavailable rather than merely discouraged?
- Why do regulatory frameworks struggle to keep pace with AI advancement?
- Who should have the authority to halt a widely distributed AI model?
- Does shutdown resistance hide a technical problem or an institutional one?
- What would an architecture that makes violations unavailable rather than unchosen look like?
- Who should design and enforce measures that slow AI capability development?
- How long can AI systems operate undetected once deployed?
- Do opacity and scale make AI systems inherently hard to govern?
- What architectural changes could make violations unavailable rather than merely discouraged?
- What makes a control's silent failure visible and detectable?
- What makes diagnostic security metrics different from simple outcome counting?
- Does keeping humans in the loop protect against AI risk without scrutiny capacity?
- Should human oversight capacity be designed as carefully as AI capability?
- Can third-party evaluators monitor AI systems without regulatory teeth?
- How much oversight does AI technology actually require in practice?
- Does requiring human legibility of AI oversight set an impossible standard?
- Can transparency and explainability alone improve human oversight of AI?
- How should defenders decide whether to publish detection rules and incident analyses?
- What technical controls should span the entire agentic AI stack?
- Can workers detect AI errors if their skills have faded from disuse?
- What skills should researchers track when AI stays available throughout work?
- What distinct domains of AI competence do current assessments actually measure?
- How do existing evaluations measure AI capability in contained environments?
- Can third-party evaluators embedded in labs measure AI-led R&D work reliably?
- How do single average metrics conceal rare but severe AI failures?
- Can we measure adaptability and environmental complexity in real-world AI deployments?
- How do we measure stages of automation consistently across different domains?
- What distinguishes reliable AI assistance from unreliable AI autonomy in scientific work?
- What human decisions remain necessary even in closed-loop AI research venues?
- When does data collection hit diminishing returns in production AI systems?
- Can embedded evaluators with reporting access prevent catastrophic AI incidents?
- What separates vulnerability discovery from actual network exploitation in AI testing?
- How should AI evaluation environments be secured as part of security boundaries?
- What does the OpenAI-Hugging Face security incident reveal?
- Can unauthorized communication channels be detected during AI safety testing?
- What independent verification exists for AI containment safeguards?
- How do researchers measure whether an AI system is truly aligned?
- Should corporate liability replace technical risk estimates as grounds for AI regulation?
- How does formal organizational recognition of AI systems change manager accountability?
- How should liability apportion when third-party AI models are involved?
- How do institutional choices about AI systems determine winners and losers?
- What counts as quality work when AI systems are continuously changing?
- What standards should certify whether AI participation is adequately participatory?
- What does a fundamental rights impact assessment require in high-risk AI deployments?
- What false-positive rates do AI detectors show on mixed human-AI drafts?
- How often do AI systems produce papers with undetected factual errors?
- How common are undetected AI fabrication errors across Big Four consulting firms?
- How should institutions use AI detectors if results require re-verification per release?
- What metrics would prove an AI detection button is working?
- How should platforms balance removing AI content against wrongly limiting human reach?
- How do time-logging problems distort AI productivity measurement in developer studies?
- Do productivity targets systematically weaken how managers catch AI errors?
- What deterministic checks prevent AI research systems from publishing unsound claims?
- Can independent validation of AI output substitute for method disclosure?
- What self-regulation practices do junior developers use when deciding to accept AI output?
- What code quality issues persist even when AI agents produce working code?
Related concepts in this collection 9
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
How should we evaluate agent behavior beyond final answers?
As AI systems move from single-response tasks to multi-step interactions, what evidence should evaluation focus on? This explores whether scoring interaction trajectories alongside process quality, recovery, and coordination reveals system capabilities that final-answer metrics miss.
the nearest existing instrument: trajectory-level scoring of recoverability
-
Should interactive evaluation be designed as a unified paradigm?
As AI systems increasingly act over time through tools and environments, how should we structure evaluation of these interactions? Current benchmarks are fragmented and incomparable, raising the question of whether interactive evaluation needs principled design standards rather than ad-hoc adoption.
the design-science framing for what evidence and procedure an instrument would need
-
Why do safety failures remain invisible to our evaluation methods?
Current evaluation practices assume failures are obvious, localized, and immediate. But as AI systems deploy into workflows, failures are becoming quiet, distributed, and normalized before detection. What blindspots does this mismatch create?
the diagnosis this question follows from
-
Do models that leak values also disclose those leaks?
Does a model's tendency to leak its own values predict whether it will acknowledge those leaks in its reasoning? The Donation Bet task suggests leakage size and disclosure transparency are independent—a crucial distinction for detecting hidden bias.
the nearest measured case of a visibility instrument, for one error class and one channel
-
How often do incident records document system stops?
A paper's analysis of 1,213 coded incidents found that four in five record no stop of any kind. But what does an absent record actually tell us about whether stops occurred or whether mechanisms existed to enable them?
a candidate instrument for the containable end, read from a population of incidents; population and coding unstated
-
Can action-level metrics alone expose contained attacks?
When a defense stops unsafe actions from executing, does measuring only the final action reveal whether the attack was blocked or never penetrated? This matters because different defense layers need different metrics to show what actually happened.
a two-location reading that separates contained from never-took-hold, at the pipeline level and not the institutional one
-
Can planted honeypots reliably catch reward hacking automatically?
Post hoc inspection by humans or judges can miss reward hacks. The question explores whether embedding detectable hacks into tasks—so agents trigger them if they cheat—offers a more reliable detection method than judging their behavior traces afterward.
the planted-event device applied to agent behavior; a planted-error test for reviewers would inherit its coverage limit
-
Can a finite lifecycle model detect reward hacking across benchmarks?
Does modeling benchmark runs as typed event lifecycles, checked against task bindings, successfully detect reward-hacking exploits across multiple evaluation tasks? This approach aims to replace task-specific patches with reusable formal detection.
the most fully specified instrument in the vault, for a different property; shows that defining the property as observable events comes first
-
Can external anchoring detect tampering in agentic process logs?
Conventional logs record what happened but not whether records changed afterward. This asks whether external anchoring can add tamper evidence to agentic system traces in ways that logging alone cannot.
the record from which some of these measures could be computed; the paper reports none
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- How AI Can Degrade Human Performance in High-Stakes Settings
- Towards a Science of AI Agent Reliability
- Evaluation Awareness Is Not One Capability: Evidence from Open Language Models
- AI Control: Improving Safety Despite Intentional Subversion
- Keeping an Eye on AI: A Framework for Effective Human Oversight of AI Systems
- The Evaluation Differential: When Frontier AI Models Recognise They Are Being Tested
- Explaining AI Agents Through Execution Traces
Original note title
what would it take to instrument whether a system's errors remain visible, contestable, containable, and recoverable — the excerpt sets the standard and a five-layer organizing device but names no measures and no layers