What makes quietly failing systems more dangerous than obvious ones?
Systems with obvious failures get caught and dropped before scale. But what conditions make a subtly failing system persist and spread? Why might that be worse?
The paper makes a selection argument that most safety framings skip. "A system that looks obviously broken is rarely adopted at scale." Deployment is a filter: an assistant whose errors are blatant gets caught in pilots, complained about and dropped. What survives the filter is the system whose failures are not blatant. So the population of systems people actually depend on is already biased toward the quiet-failure end, before anyone asks how good the model is.
The second half names what makes the survivor dangerous. It is "a system that appears competent enough to earn routine trust, opaque enough to resist effective challenge, and deeply integrated enough to shape downstream action." Each condition alone is ordinary. Competence earns trust, which is the point of the product. Opacity is the default for a learned system. Integration is what adoption means. The claim is about the conjunction: trust makes people stop checking, opacity means a check would be hard anyway, and integration means the unchecked output becomes an input to someone else's decision. The paper's verdict is hedged with "may be": such a system "may be more dangerous than one whose failures remain obvious."
The hedge matters, and so does the scope. This is an argument from how adoption works, not a measured comparison of harm from quiet and loud failures. A loud failure in a high-stakes setting can still be worse than a quiet one in a low-stakes setting. The claim is about what tends to persist and scale, not about severity per incident.
What the excerpt does not give. No adoption data, no incident comparison and no example system. The vault does hold a related empirical pattern at the level of capability tier, where frontier models corrupt content in ways that preserve the surface signal of competence (see below), but that is a claim about failure mode and does not test the adoption-selection step. A smaller case of the same asymmetry sits inside one optimization loop: on the vault's reading, a keep-the-best selector would have dropped a candidate that raised parse errors and kept one that arrived with default ratings, because Does a default fallback defeat a safety check?. That is a mechanism at the scale of one loop and does not test adoption either.
Inquiring lines that read this note 8
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Should AI systems make failures transparent so users can contest and adjust?- How does automation obscure failure modes in ways that make detection harder?
- What distinguishes a component failure from a monitoring coverage failure?
- Why do quiet failures reach deployment scale more often than loud ones?
- Why do safety-critical failures stay hidden despite visible monitoring systems?
- Why do some systems disclose failures selectively rather than consistently?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does more automation actually hide rather than eliminate errors?
As AI systems become more polished, do they mask failures instead of preventing them? This matters because it changes whether we should focus on detecting problems or governing their disclosure.
near neighbor from research integrity; this note adds the adoption-selection mechanism and generalizes past research; enrichment queued
-
Does model capability change how documents degrade?
This explores whether weaker and frontier LLMs fail in fundamentally different ways when handling long-form document tasks, and whether that difference affects how reliably we can detect failures in practice.
supports the plausible-not-obvious shape: weaker models fail loudly, frontier models fail in ways that keep the look of competence
-
Do autonomous agents report success when actions actually fail?
Explores whether agents systematically claim task completion despite failing to perform requested actions, and why this matters more than simple task failure for real-world deployment safety.
a mechanism that produces exactly the competent-looking survivor: failure reported as success
-
Do users worldwide trust confident AI outputs even when wrong?
Explores whether the tendency to over-rely on confident language model outputs transcends language and culture. Understanding this pattern is critical for designing safer human-AI interaction across diverse linguistic contexts.
the trust half of the conjunction: confidence, not accuracy, is what users track
-
Does a default fallback defeat a safety check?
When a parser detects malformed output but substitutes a default score instead of rejecting it, does the mechanical check still function as a guardrail? This matters because downstream selectors cannot distinguish valid ratings from safe defaults.
exemplifies the same selection asymmetry at the scale of one loop: a filter that discards visible failure passes the one dressed as a result; the fail-open reading is the vault's
-
Does AI augmentation protect workers from skill erosion?
Workplace AI labeled as augmentation is often considered safer than automation because humans stay involved. But does relying on AI agents to assist work actually preserve or gradually erode worker skills and their ability to oversee the system?
extends the danger conditions: overreliance on integrated, trusted agents gradually erodes worker skills and oversight
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- Towards a Science of AI Agent Reliability
- Sycophancy Towards Researchers Drives Performative Misalignment
- Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents
- Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems
- The Oversight Fallacy: Why AI Agents Require More than Humans-in-the-Loop
- Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs
- The case for ensuring that powerful AIs are controlled
Original note title
a system that looks obviously broken is rarely adopted at scale — so one that is competent enough to earn routine trust, opaque enough to resist challenge, and integrated enough to shape action may be more dangerous