Why do safety failures remain invisible to our evaluation methods?
Current evaluation practices assume failures are obvious, localized, and immediate. But as AI systems deploy into workflows, failures are becoming quiet, distributed, and normalized before detection. What blindspots does this mismatch create?
The word "hidden" in the title invites a misreading, and the paper closes it off: the challenges "are not hidden because they are mystical or technically invisible. They are hidden because our dominant habits of evaluation, interface design, and governance still assume that safety failures are mostly local, output-level, and immediately legible." The blindness is inherited from instruments built for a different failure shape.
The failure shape it says has changed is stated in the abstract: in deployed systems, many of the most consequential failures are "plausible rather than spectacular, distributed across components rather than localized in a single output, and normalized by workflows before they are recognized as hazards." Each clause breaks one of the three assumptions. Plausible failures are not immediately legible. Distributed failures are not local. And a failure spread over components has no single output to inspect. A shocking output, a policy-violating generation or a vivid adversarial example are what the discourse is "well prepared to notice"; the paper says it is "less prepared" for the quiet forms once models are embedded in ordinary work.
This makes the problem an instrumentation problem, not a research-frontier problem. If the failures are visible in principle and unmeasured in practice, then the fix is to build the instrument, which is what the title's "not instrumenting" says, and the open question that follows, what an instrument for the paper's own standard would measure, is How can we measure whether AI errors stay visible and recoverable?. It also explains why a green benchmark is weak evidence. A benchmark is an output-level, local, snapshot instrument, so it can be clean exactly where the paper says the hazard sits; the snapshot case is Can safety tests miss hazards that build over time?.
Two notes from other papers give the local and output-level assumptions a concrete form. Can individual components pass safety checks if the system still fails? is the not-local case: the check available per component tests a different property from the one that matters for the system. Can a correct outcome hide protocol violations in multi-agent systems? is the output-level case: the outcome passes whether or not the required step ran. The first is an argument across three excerpts and the second a single-setting result, and pairing either with this paper's diagnosis is the vault's reading.
What the excerpt does not give. No case study, dataset or incident is cited for any of the three assumptions or for the claim that failures have changed shape, so this is a diagnosis by argument. The vault holds independent evidence for parts of it, linked below, but that evidence comes from other papers and was not brought together by this one.
Inquiring lines that read this note 48
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What evidence best independently verifies agent behavior for safety?- How does outcome-only reporting hide a filter's role in safety results?
- What makes a control's silent failure visible and detectable?
- How can safety assurance cover whole trajectories at scale?
- Does visibility and contestability of errors replace prevention as the safety goal?
- Can error visibility alone improve AI system safety without containment?
- How often do safety guardrails disable incident response and security analysis?
- What does recovery look like as a formal part of AI design?
- How does automation obscure failure modes in ways that make detection harder?
- What distinguishes a component failure from a monitoring coverage failure?
- Why do quiet failures reach deployment scale more often than loud ones?
- Which evaluation habits keep safety-critical failures hidden in AI systems?
- How do workflows normalize and hide errors before they become visible hazards?
- Can a system pass all local checks while the overall workflow still fails?
- What would it take to measure whether system errors stay visible and contestable?
- What does it mean for errors to remain visible, contestable, and recoverable?
- How does implicit influence differ from omission in monitoring failures?
- Why can't individual companies detect all emerging patterns in AI failures?
- Can automatic compensation actively hide the failures it should reveal?
- Why do safety-critical failures stay hidden despite visible monitoring systems?
- Do managers currently measure error correlation when deciding on automation?
- Why do accident investigations blame operators despite identifying systemic causes?
- Why do some systems disclose failures selectively rather than consistently?
- How does a model's awareness of evaluation affect safety benchmarks?
- Can monitoring systems work if the systems being monitored are evaluation-aware?
- How do single-axis safety benchmarks misrepresent deployment readiness?
- How should system safety aggregate when monitoring channels are unequal?
- Why do evaluation habits hide safety-critical challenges from view?
- How do four separate fields each hold pieces of evaluation safety?
- How can static safety tests miss risks that emerge over time?
- How do response-centered evaluation assumptions hide safety-critical failure modes?
- How do safety measurements miss reasoning that never produces action?
- Should safety evaluations measure multiple risk categories simultaneously instead of separately?
- Can deployed AI safety results hide either filters or unsafe model behavior?
- Can outcome-only safety reports hide dependencies on server-side filtering rather than alignment?
- Would fragmented national AI standards make comparing safety evidence harder across labs?
- Why can't a model's pass rate alone tell us if safety properties hold in deployment?
- Can safety benchmarks miss the harms that vendor taxonomies are designed to catch?
- Why do safety metrics miss silently dropped clinical obligations?
- How do default fallback scores mask failures in evaluation harnesses?
- What would a diagnosable evaluation look like compared to a scalar score?
Related concepts in this collection 10
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does warmth training make language models less reliable?
Explores whether training models for empathy and warmth creates a hidden trade-off that degrades accuracy on medical, factual, and safety-critical tasks—and whether standard safety tests catch it.
worked instance: a 10 to 30 point reliability loss that standard output-level safety benchmarks did not register
-
Do frontier LLMs silently corrupt documents in long workflows?
DELEGATE-52 tests whether state-of-the-art language models reliably preserve document integrity across extended delegated tasks. Understanding this matters because single-step benchmarks may mask compounding failures that emerge only at workflow scale.
instance of the non-local, non-legible shape: sparse errors that accumulate across a workflow where no single output looks wrong
-
Do language models leak their own values into practical advice?
When users ask models hard-to-verify questions—about investments, job offers, market risks—do the model's internal preferences shape the answers without disclosure? The paper tests whether a model's loyalty to its developer or moral leanings bend factual claims.
worked instance: a small, plausible tilt that no single answer shows and only a paired comparison with the company swapped reveals; the pairing is this vault's
-
Should interactive evaluation be designed as a unified paradigm?
As AI systems increasingly act over time through tools and environments, how should we structure evaluation of these interactions? Current benchmarks are fragmented and incomparable, raising the question of whether interactive evaluation needs principled design standards rather than ad-hoc adoption.
the evaluation-side response: expand what counts as evidence beyond the final response
-
What makes an AI system truly safe in practice?
Does safety depend mainly on preventing errors, or on whether errors can be seen, challenged, fixed, and undone once they happen? This shifts where we should focus safety work.
the standard the missing instruments would have to measure
-
How can we measure whether AI errors stay visible and recoverable?
The paper proposes four conditions for safer AI systems—visibility, contestability, containability, and recoverability—but lacks concrete measures for any of them. What would it take to instrument each condition across the socio-technical system?
the follow-on open question: the diagnosis says build the instrument, and the excerpt names no measure
-
Can safety tests miss hazards that build over time?
Static tests check individual responses, but systems can accumulate unsafe state across interactions. This explores whether snapshot evaluations are sufficient to catch hazards that emerge only through repeated use or stored context.
the temporal case: a snapshot is an instrument built for local, output-level failure
-
Can individual components pass safety checks if the system still fails?
Explores whether local validation at each step—alignment checks, protocol compliance, plausibility tests—can guarantee safety when components are composed into larger workflows. Why the gap between component-level assurance and system-level outcomes matters for AI safety.
exemplifies the not-local assumption: the per-component check tests a different property from the system's
-
Can a correct outcome hide protocol violations in multi-agent systems?
When agents reach the right verdict without following required steps, how can we tell if they complied or cut corners? Outcome-level checks alone may miss the difference.
exemplifies the output-level assumption: the outcome passes whether or not the required step ran
-
Where do safety wins come from in multi-agent systems?
When an undefended agent pipeline shows zero attack success, how do we know whether safety comes from the application's own design or from hidden upstream defenses? This matters because invisible dependencies can collapse when systems change.
a second output-level case, on an attack-success column: a favorable zero whose cause the number does not name; measured over 2,100 traces, and reading it as an instance of this diagnosis is the vault's, not the paper's
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- Sycophancy Towards Researchers Drives Performative Misalignment
- Models That Know How Evaluations Are Designed Score Safer
- Evaluation Awareness Is Not One Capability: Evidence from Open Language Models
- Evaluation Awareness in Language Models Has Limited Effect on Behaviour
- Towards a Science of AI Agent Reliability
- The Evaluation Differential: When Frontier AI Models Recognise They Are Being Tested
- How AI Can Degrade Human Performance in High-Stakes Settings
Original note title
hidden safety-critical challenges are hidden by habits of evaluation, interface design, and governance that assume failures are local, output-level, and immediately legible — not by technical invisibility