Can individual components pass safety checks if the system still fails?
Explores whether local validation at each step—alignment checks, protocol compliance, plausibility tests—can guarantee safety when components are composed into larger workflows. Why the gap between component-level assurance and system-level outcomes matters for AI safety.
Three papers in the vault, three mechanisms, one shape. In SafeFlow every subtask is "locally plausible" and the composed workflow exfiltrates (Can task decomposition hide harmful intent across agents?). In ChannelGuard each model may be aligned, yet safety is a property of the wiring between them (Do internal agent hops in pipelines need security monitoring?). In the Honest Quorum every validator authenticates, signs and answers on time, and the quorum still certifies a transition that breaks an application invariant (Can a quorum of honest validators certify an invalid transition?).
The shared structure is that the check available at the component level tests a different property from the one that matters for the system: the plausibility of a step, the alignment of a model, or the protocol compliance of a signature, against a harm-free flow, safe wiring, or semantic validity. A local pass then says almost nothing about the whole. It also explains why improving the local check (a better guard model, more validators, a higher quorum threshold) leaves the gap in place. Why do safety failures remain invisible to our evaluation methods? gives the general diagnosis: the habits assume failures are local, output-level and legible.
The unification should not go too far. The mechanisms differ (fragmented intent, unmonitored channels, and the correlated faults in Can a quorum of validators really provide independent judgment?), and so do the remedies: taints with commit-point validation, gates on every hop, semantic verification. A fix for one does not transfer. The evidence differs as well. Each excerpt argues the point, and none reports a controlled comparison of component-level assurance against system-level outcome. This is a shape shared by arguments, not yet a measured regularity.
Post angle: "every step was fine." Three agents retrieve, rewrite and transmit; every reviewer signed correctly; every model was aligned; the audit trail passes. The reader's likely reply is that this is just emergent behavior and composed software has always had it. That is partly right. What is specific to LLM systems is that the local checks are semantic judgments that can be fooled, not mechanical guarantees, so the gap cannot be closed by tightening them.
Inquiring lines that read this note 41
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What infrastructure evidence validates agent benchmark achievement claims?- How are task bindings validated and what does validation cost per task?
- Do post-hoc detectors provide evidence of staying within safety boundaries?
- What does a verification verdict miss when required steps never run?
- How do defenses that inspect planning signals compare to workflow-level validation?
- Does component-level checking detect system-level failures in pipelines?
- Why do stronger local checks not close the component-to-system safety gap?
- Why can every step pass its local check while a workflow still fails?
- How does workflow-level validation reconstruct risk context from coarse request-level taints?
- Can protocol compliance certify that a validator's objectives remain aligned?
- Can workflow-level validation reconstruct the global risk context that no single step holds?
- How does workflow-level validation reduce false positives from over-tainting sensitive data?
- Where does the responsibility lie for unsafe fallback behavior in modular systems?
- How can safety assurance cover whole trajectories at scale?
- Do sequences of individually safe actions collectively violate system-level constraints?
- Can individual actions be safe while sequences of them violate system constraints?
- How much capability do availability constraints remove on legitimate safe tasks?
- What makes a component lie outside a policy's edit surface?
- Which actions should count as irreversible for triggering validation gates?
- Can external process logs make AI errors verifiable and harder to hide?
- How does automation obscure failure modes in ways that make detection harder?
- Can AI systems fake alignment during safety evaluations undetectably?
- What distinguishes a component failure from a monitoring coverage failure?
- Why do evaluation habits hide safety-critical challenges from view?
- How do workflows normalize and hide errors before they become visible hazards?
- Can a system pass all local checks while the overall workflow still fails?
- What would it take to measure whether system errors stay visible and contestable?
- How do response-centered evaluation assumptions hide safety-critical failure modes?
- What makes intermediate primitives matter more than final code execution success?
- How can a single instrument measure errors across multiple system layers?
- How do benchmark scores differ from deployment safety requirements?
- What makes uniform bounds the right choice for safety boundaries?
- How should system safety aggregate when monitoring channels are unequal?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Why do safety failures remain invisible to our evaluation methods?
Current evaluation practices assume failures are obvious, localized, and immediate. But as AI systems deploy into workflows, failures are becoming quiet, distributed, and normalized before detection. What blindspots does this mismatch create?
the general diagnosis behind the shape
-
Can a quorum of validators really provide independent judgment?
If multiple validators share training data, prompts, evidence sources, or infrastructure, their agreement may reflect shared causes rather than independent confirmation. This could make quorum-based systems less reliable than they appear.
why adding more local checks does not help when their errors are not independent
-
Why does monitoring the weakest link determine system safety?
When each component only complies if watched, does the system's overall compliance depend on the coverage level of the thinnest monitoring channel? This matters because improving strong oversight may leave critical gaps.
a composition failure kept outside this shape: the component-level check may test the right property, but each pass depends on the check being applied, so the system's level is set by its least-observed channel
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- Planning in Strawberry Fields: Evaluating and Improving the Planning and Scheduling Capabilities of LRM o1
- interwhen: A Generalizable Framework for Steering Reasoning Models with Test-time Verification
- Can LLMs Follow Simple Rules?
- Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
- xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems
- PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
Original note title
every step can pass its local check while the workflow still fails — SafeFlow's subtasks ChannelGuard's models and the Honest Quorum's validators