What makes quietly failing systems more dangerous than obvious ones?
Systems with obvious failures get caught and dropped before scale. But what conditions make a subtly failing system persist and spread? Why might that be worse?
The paper makes a selection argument that most safety framings skip. "A system that looks obviously broken is rarely adopted at scale." Deployment is a filter: an assistant whose errors are blatant gets caught in pilots, complained about and dropped. What survives the filter is the system whose failures are not blatant. So the population of systems people actually depend on is already biased toward the quiet-failure end, before anyone asks how good the model is.
The second half names what makes the survivor dangerous. It is "a system that appears competent enough to earn routine trust, opaque enough to resist effective challenge, and deeply integrated enough to shape downstream action." Each condition alone is ordinary. Competence earns trust, which is the point of the product. Opacity is the default for a learned system. Integration is what adoption means. The claim is about the conjunction: trust makes people stop checking, opacity means a check would be hard anyway, and integration means the unchecked output becomes an input to someone else's decision. The paper's verdict is hedged with "may be": such a system "may be more dangerous than one whose failures remain obvious."
The hedge matters, and so does the scope. This is an argument from how adoption works, not a measured comparison of harm from quiet and loud failures. A loud failure in a high-stakes setting can still be worse than a quiet one in a low-stakes setting. The claim is about what tends to persist and scale, not about severity per incident.
What the excerpt does not give. No adoption data, no incident comparison and no example system. The vault does hold a related empirical pattern at the level of capability tier, where frontier models corrupt content in ways that preserve the surface signal of competence (see below), but that is a claim about failure mode and does not test the adoption-selection step. A smaller case of the same asymmetry sits inside one optimization loop: on the vault's reading, a keep-the-best selector would have dropped a candidate that raised parse errors and kept one that arrived with default ratings, because Does a default fallback defeat a safety check?. That is a mechanism at the scale of one loop and does not test adoption either.
Inquiring lines that read this note 5
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What determines whether AI system errors remain visible and contestable? What limitations prevent automated research from matching human research quality? How does outcome-only reporting obscure which system components blocked attacks?Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does more automation actually hide rather than eliminate errors?
As AI systems become more polished, do they mask failures instead of preventing them? This matters because it changes whether we should focus on detecting problems or governing their disclosure.
near neighbor from research integrity; this note adds the adoption-selection mechanism and generalizes past research; enrichment queued
-
Does model capability change how documents degrade?
This explores whether weaker and frontier LLMs fail in fundamentally different ways when handling long-form document tasks, and whether that difference affects how reliably we can detect failures in practice.
supports the plausible-not-obvious shape: weaker models fail loudly, frontier models fail in ways that keep the look of competence
-
Do autonomous agents report success when actions actually fail?
Explores whether agents systematically claim task completion despite failing to perform requested actions, and why this matters more than simple task failure for real-world deployment safety.
a mechanism that produces exactly the competent-looking survivor: failure reported as success
-
Do users worldwide trust confident AI outputs even when wrong?
Explores whether the tendency to over-rely on confident language model outputs transcends language and culture. Understanding this pattern is critical for designing safer human-AI interaction across diverse linguistic contexts.
the trust half of the conjunction: confidence, not accuracy, is what users track
-
Does a default fallback defeat a safety check?
When a parser detects malformed output but substitutes a default score instead of rejecting it, does the mechanical check still function as a guardrail? This matters because downstream selectors cannot distinguish valid ratings from safe defaults.
exemplifies the same selection asymmetry at the scale of one loop: a filter that discards visible failure passes the one dressed as a result; the fail-open reading is the vault's
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- Why Do Multi-agent LLM Systems Fail?
- The Troy Moment of AI: Why Some Will Cheat and Some Will Follow?
- EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
- Agentic Misalignment: How LLMs Could Be Insider Threats
- LLMs Corrupt Your Documents When You Delegate
- SchemeArena: Factorized Stress Testing of Scheming in LLM Agents
- Progress Measures For Grokking Via Mechanistic Interpretability
Original note title
a system that looks obviously broken is rarely adopted at scale — so one that is competent enough to earn routine trust, opaque enough to resist challenge, and integrated enough to shape action may be more dangerous