Why do safety failures remain invisible to our evaluation methods?
Current evaluation practices assume failures are obvious, localized, and immediate. But as AI systems deploy into workflows, failures are becoming quiet, distributed, and normalized before detection. What blindspots does this mismatch create?
The word "hidden" in the title invites a misreading, and the paper closes it off: the challenges "are not hidden because they are mystical or technically invisible. They are hidden because our dominant habits of evaluation, interface design, and governance still assume that safety failures are mostly local, output-level, and immediately legible." The blindness is inherited from instruments built for a different failure shape.
The failure shape it says has changed is stated in the abstract: in deployed systems, many of the most consequential failures are "plausible rather than spectacular, distributed across components rather than localized in a single output, and normalized by workflows before they are recognized as hazards." Each clause breaks one of the three assumptions. Plausible failures are not immediately legible. Distributed failures are not local. And a failure spread over components has no single output to inspect. A shocking output, a policy-violating generation or a vivid adversarial example are what the discourse is "well prepared to notice"; the paper says it is "less prepared" for the quiet forms once models are embedded in ordinary work.
This makes the problem an instrumentation problem, not a research-frontier problem. If the failures are visible in principle and unmeasured in practice, then the fix is to build the instrument, which is what the title's "not instrumenting" says, and the open question that follows, what an instrument for the paper's own standard would measure, is How can we measure whether AI errors stay visible and recoverable?. It also explains why a green benchmark is weak evidence. A benchmark is an output-level, local, snapshot instrument, so it can be clean exactly where the paper says the hazard sits; the snapshot case is Can safety tests miss hazards that build over time?.
Two notes from other papers give the local and output-level assumptions a concrete form. Can individual components pass safety checks if the system still fails? is the not-local case: the check available per component tests a different property from the one that matters for the system. Can a correct outcome hide protocol violations in multi-agent systems? is the output-level case: the outcome passes whether or not the required step ran. The first is an argument across three excerpts and the second a single-setting result, and pairing either with this paper's diagnosis is the vault's reading.
What the excerpt does not give. No case study, dataset or incident is cited for any of the three assumptions or for the claim that failures have changed shape, so this is a diagnosis by argument. The vault holds independent evidence for parts of it, linked below, but that evidence comes from other papers and was not brought together by this one.
Inquiring lines that read this note 26
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How does outcome-only reporting obscure which system components blocked attacks?- How does outcome-only reporting hide a filter's role in safety results?
- How do benchmark scores differ from deployment safety requirements?
- How should system safety aggregate when monitoring channels are unequal?
- What makes a control's silent failure visible and detectable?
- How do default fallback scores mask failures in evaluation harnesses?
- How do safety measurements miss reasoning that never produces action?
- What would a diagnosable evaluation look like compared to a scalar score?
- What does recovery look like as a formal part of AI design?
- How does automation obscure failure modes in ways that make detection harder?
- Does visibility and contestability of errors replace prevention as the safety goal?
- What distinguishes a component failure from a monitoring coverage failure?
- Why do quiet failures reach deployment scale more often than loud ones?
- Which evaluation habits keep safety-critical failures hidden in AI systems?
- Why do evaluation habits hide safety-critical challenges from view?
- How do workflows normalize and hide errors before they become visible hazards?
- Can a system pass all local checks while the overall workflow still fails?
- What would it take to measure whether system errors stay visible and contestable?
- How do response-centered evaluation assumptions hide safety-critical failure modes?
- What does it mean for errors to remain visible, contestable, and recoverable?
- How do inherited evaluation habits obscure failures that matter most?
Related concepts in this collection 10
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does warmth training make language models less reliable?
Explores whether training models for empathy and warmth creates a hidden trade-off that degrades accuracy on medical, factual, and safety-critical tasks—and whether standard safety tests catch it.
worked instance: a 10 to 30 point reliability loss that standard output-level safety benchmarks did not register
-
Do frontier LLMs silently corrupt documents in long workflows?
DELEGATE-52 tests whether state-of-the-art language models reliably preserve document integrity across extended delegated tasks. Understanding this matters because single-step benchmarks may mask compounding failures that emerge only at workflow scale.
instance of the non-local, non-legible shape: sparse errors that accumulate across a workflow where no single output looks wrong
-
Do language models leak their own values into practical advice?
When users ask models hard-to-verify questions—about investments, job offers, market risks—do the model's internal preferences shape the answers without disclosure? The paper tests whether a model's loyalty to its developer or moral leanings bend factual claims.
worked instance: a small, plausible tilt that no single answer shows and only a paired comparison with the company swapped reveals; the pairing is this vault's
-
Should interactive evaluation be designed as a unified paradigm?
As AI systems increasingly act over time through tools and environments, how should we structure evaluation of these interactions? Current benchmarks are fragmented and incomparable, raising the question of whether interactive evaluation needs principled design standards rather than ad-hoc adoption.
the evaluation-side response: expand what counts as evidence beyond the final response
-
What makes an AI system truly safe in practice?
Does safety depend mainly on preventing errors, or on whether errors can be seen, challenged, fixed, and undone once they happen? This shifts where we should focus safety work.
the standard the missing instruments would have to measure
-
How can we measure whether AI errors stay visible and recoverable?
The paper proposes four conditions for safer AI systems—visibility, contestability, containability, and recoverability—but lacks concrete measures for any of them. What would it take to instrument each condition across the socio-technical system?
the follow-on open question: the diagnosis says build the instrument, and the excerpt names no measure
-
Can safety tests miss hazards that build over time?
Static tests check individual responses, but systems can accumulate unsafe state across interactions. This explores whether snapshot evaluations are sufficient to catch hazards that emerge only through repeated use or stored context.
the temporal case: a snapshot is an instrument built for local, output-level failure
-
Can individual components pass safety checks if the system still fails?
Explores whether local validation at each step—alignment checks, protocol compliance, plausibility tests—can guarantee safety when components are composed into larger workflows. Why the gap between component-level assurance and system-level outcomes matters for AI safety.
exemplifies the not-local assumption: the per-component check tests a different property from the system's
-
Can a correct outcome hide protocol violations in multi-agent systems?
When agents reach the right verdict without following required steps, how can we tell if they complied or cut corners? Outcome-level checks alone may miss the difference.
exemplifies the output-level assumption: the outcome passes whether or not the required step ran
-
Where do safety wins come from in multi-agent systems?
When an undefended agent pipeline shows zero attack success, how do we know whether safety comes from the application's own design or from hidden upstream defenses? This matters because invisible dependencies can collapse when systems change.
a second output-level case, on an attack-success column: a favorable zero whose cause the number does not name; measured over 2,100 traces, and reading it as an instance of this diagnosis is the vault's, not the paper's
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- Open-World Evaluations for Measuring Frontier AI Capabilities
- Explaining AI Agents Through Execution Traces
- Tell me about yourself: LLMs are aware of their learned behaviors
- Large Language Model Reasoning Failures
- Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
- Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
- Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Original note title
hidden safety-critical challenges are hidden by habits of evaluation, interface design, and governance that assume failures are local, output-level, and immediately legible — not by technical invisibility