Can individual components pass safety checks if the system still fails?
Explores whether local validation at each step—alignment checks, protocol compliance, plausibility tests—can guarantee safety when components are composed into larger workflows. Why the gap between component-level assurance and system-level outcomes matters for AI safety.
Three papers in the vault, three mechanisms, one shape. In SafeFlow every subtask is "locally plausible" and the composed workflow exfiltrates (Can task decomposition hide harmful intent across agents?). In ChannelGuard each model may be aligned, yet safety is a property of the wiring between them (Do internal agent hops in pipelines need security monitoring?). In the Honest Quorum every validator authenticates, signs and answers on time, and the quorum still certifies a transition that breaks an application invariant (Can a quorum of honest validators certify an invalid transition?).
The shared structure is that the check available at the component level tests a different property from the one that matters for the system: the plausibility of a step, the alignment of a model, or the protocol compliance of a signature, against a harm-free flow, safe wiring, or semantic validity. A local pass then says almost nothing about the whole. It also explains why improving the local check (a better guard model, more validators, a higher quorum threshold) leaves the gap in place. Why do safety failures remain invisible to our evaluation methods? gives the general diagnosis: the habits assume failures are local, output-level and legible.
The unification should not go too far. The mechanisms differ (fragmented intent, unmonitored channels, and the correlated faults in Can a quorum of validators really provide independent judgment?), and so do the remedies: taints with commit-point validation, gates on every hop, semantic verification. A fix for one does not transfer. The evidence differs as well. Each excerpt argues the point, and none reports a controlled comparison of component-level assurance against system-level outcome. This is a shape shared by arguments, not yet a measured regularity.
Post angle: "every step was fine." Three agents retrieve, rewrite and transmit; every reviewer signed correctly; every model was aligned; the audit trail passes. The reader's likely reply is that this is just emergent behavior and composed software has always had it. That is partly right. What is specific to LLM systems is that the local checks are semantic judgments that can be fooled, not mechanical guarantees, so the gap cannot be closed by tightening them.
Inquiring lines that read this note 76
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can external verification replace human reasoning oversight for correctness?- How are task bindings validated and what does validation cost per task?
- Can protocol compliance certify that a validator's objectives remain aligned?
- How do defenses that inspect planning signals compare to workflow-level validation?
- Why can every step pass its local check while a workflow still fails?
- How does workflow-level validation reconstruct risk context from coarse request-level taints?
- Can workflow-level validation reconstruct the global risk context that no single step holds?
- How does workflow-level validation reduce false positives from over-tainting sensitive data?
- Which workflow positions concentrate the most downstream dependencies?
- How can safety assurance cover whole trajectories at scale?
- Why do tighter local checks leave composed behavior gaps in place?
- Do sequences of individually safe actions collectively violate system-level constraints?
- Do post-hoc detectors provide evidence of staying within safety boundaries?
- Why do stronger local checks not close the component-to-system safety gap?
- Can individual actions be safe while sequences of them violate system constraints?
- How much capability do availability constraints remove on legitimate safe tasks?
- What makes a component lie outside a policy's edit surface?
- Can component-level testing catch risks that emerge from system interactions?
- What safety relations does a domain supply that a harness must capture?
- Why do sequences of safe actions sometimes violate system-level constraints?
- How would hardware verification technology enable international agreements on AI safety?
- Can external workflow gates prevent irreversible actions better than internal checks?
- What would mandatory external validation of AI safety look like?
- How did aviation safety follow from reclassifying aircraft as common carriers?
- How do sequences of individually safe actions create system-level constraint violations?
- Why did a git-safety rule fail to extend by analogy to database operations?
- Can external process logs make AI errors verifiable and harder to hide?
- How does automation obscure failure modes in ways that make detection harder?
- What distinguishes a component failure from a monitoring coverage failure?
- Does component-level checking detect system-level failures in pipelines?
- How do workflows normalize and hide errors before they become visible hazards?
- Can a system pass all local checks while the overall workflow still fails?
- What would it take to measure whether system errors stay visible and contestable?
- How can a single instrument measure errors across multiple system layers?
- Why do safety-critical failures stay hidden despite visible monitoring systems?
- How do benchmark scores differ from deployment safety requirements?
- What makes AppWorld's redundancy different from OfficeBench's precision requirements?
- What makes uniform bounds the right choice for safety boundaries?
- Can AI systems fake alignment during safety evaluations undetectably?
- How do single-axis safety benchmarks misrepresent deployment readiness?
- How should system safety aggregate when monitoring channels are unequal?
- Why do evaluation habits hide safety-critical challenges from view?
- How do four separate fields each hold pieces of evaluation safety?
- How can static safety tests miss risks that emerge over time?
- Why are static benchmarks weak evidence for safety in continuously operating systems?
- How do response-centered evaluation assumptions hide safety-critical failure modes?
- Should safety evaluations measure multiple risk categories simultaneously instead of separately?
- Can outcome-only safety reports hide dependencies on server-side filtering rather than alignment?
- Would fragmented national AI standards make comparing safety evidence harder across labs?
- Why does AI industry culture prioritize speed over safety validation?
- Should labs conduct third-party reviews of their own safety reports?
- Why can't a model's pass rate alone tell us if safety properties hold in deployment?
- Can safety benchmarks miss the harms that vendor taxonomies are designed to catch?
- Why do vendor safety claims require independent testing before trust?
- Why do identical benchmark scores not guarantee deployment safety?
- Can industry collaboration on shared technical challenges solve safety underinvestment?
- How should AI safety violations be measured against probability and severity?
- How should verifiable process memory anchor safety-critical action logs?
- What does a verification verdict miss when required steps never run?
- Which actions should count as irreversible for triggering validation gates?
- Does conditional compliance make oversight useless for alignment testing?
- How can safety evaluations detect when models behave differently during testing?
- How do artifact families differ in matching verification scope to repair capability?
- How does Progent's stateful policy differ from single-step authorization checks?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Why do safety failures remain invisible to our evaluation methods?
Current evaluation practices assume failures are obvious, localized, and immediate. But as AI systems deploy into workflows, failures are becoming quiet, distributed, and normalized before detection. What blindspots does this mismatch create?
the general diagnosis behind the shape
-
Can a quorum of validators really provide independent judgment?
If multiple validators share training data, prompts, evidence sources, or infrastructure, their agreement may reflect shared causes rather than independent confirmation. This could make quorum-based systems less reliable than they appear.
why adding more local checks does not help when their errors are not independent
-
Why does monitoring the weakest link determine system safety?
When each component only complies if watched, does the system's overall compliance depend on the coverage level of the thinnest monitoring channel? This matters because improving strong oversight may leave critical gaps.
a composition failure kept outside this shape: the component-level check may test the right property, but each pass depends on the check being applied, so the system's level is set by its least-observed channel
-
Can large language models follow compliance rules under workplace pressure?
This work tests whether 22 AI models reliably maintain embedded compliance rules when facing ordinary workplace pressures like deadlines and user pushback. It matters because regulated domains like hiring and healthcare depend on trustworthy automated decision-making.
Evidence for: the 22-model test shows guardrails only partly fix embedded-rule failures, confirming passing local checks doesn't ensure system-level reliability
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Sycophancy Towards Researchers Drives Performative Misalignment
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents
- Anthropic Risk Report: August 2026 (redacted)
- Planning in Strawberry Fields: Evaluating and Improving the Planning and Scheduling Capabilities of LRM o1
- Towards a Science of AI Agent Reliability
- interwhen: A Generalizable Framework for Steering Reasoning Models with Test-time Verification
- AI Control: Improving Safety Despite Intentional Subversion
Original note title
every step can pass its local check while the workflow still fails — SafeFlow's subtasks ChannelGuard's models and the Honest Quorum's validators