SYNTHESIS NOTE
Topics›Agents Multi Architecture›this note

Can individual components pass safety checks if the system still fails?

Explores whether local validation at each step—alignment checks, protocol compliance, plausibility tests—can guarantee safety when components are composed into larger workflows. Why the gap between component-level assurance and system-level outcomes matters for AI safety.

Synthesis note · 2026-09-23 · sourced from Agents Multi Architecture

Three papers in the vault, three mechanisms, one shape. In SafeFlow every subtask is "locally plausible" and the composed workflow exfiltrates (Can task decomposition hide harmful intent across agents?). In ChannelGuard each model may be aligned, yet safety is a property of the wiring between them (Do internal agent hops in pipelines need security monitoring?). In the Honest Quorum every validator authenticates, signs and answers on time, and the quorum still certifies a transition that breaks an application invariant (Can a quorum of honest validators certify an invalid transition?).

The shared structure is that the check available at the component level tests a different property from the one that matters for the system: the plausibility of a step, the alignment of a model, or the protocol compliance of a signature, against a harm-free flow, safe wiring, or semantic validity. A local pass then says almost nothing about the whole. It also explains why improving the local check (a better guard model, more validators, a higher quorum threshold) leaves the gap in place. Why do safety failures remain invisible to our evaluation methods? gives the general diagnosis: the habits assume failures are local, output-level and legible.

The unification should not go too far. The mechanisms differ (fragmented intent, unmonitored channels, and the correlated faults in Can a quorum of validators really provide independent judgment?), and so do the remedies: taints with commit-point validation, gates on every hop, semantic verification. A fix for one does not transfer. The evidence differs as well. Each excerpt argues the point, and none reports a controlled comparison of component-level assurance against system-level outcome. This is a shape shared by arguments, not yet a measured regularity.

Post angle: "every step was fine." Three agents retrieve, rewrite and transmit; every reviewer signed correctly; every model was aligned; the audit trail passes. The reader's likely reply is that this is just emergent behavior and composed software has always had it. That is partly right. What is specific to LLM systems is that the local checks are semantic judgments that can be fooled, not mechanical guarantees, so the gap cannot be closed by tightening them.

Inquiring lines that read this note 41

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

What infrastructure evidence validates agent benchmark achievement claims? Where do unmonitored channels leave multi-agent planning vulnerable to attack? How can workflow-level validation detect semantic corruption that protocol compliance misses? How do coordinated agent sequences violate constraints that individual actions respect? What determines whether AI system errors remain visible and contestable? How does outcome-only reporting obscure which system components blocked attacks? How can we verify agent claims against their actual capabilities and actions? Can defenses detect attacks composed across multiple skills? Do AI capability benchmarks accurately measure reasoning ability or just surface patterns? Can human oversight effectively constrain capable AI agents? How can evaluations detect conditional compliance in monitored AI systems? How do evaluation methodologies affect which model capabilities are revealed or hidden? How does training data contamination persist through safety alignment mechanisms? How does position in multi-agent workflows amplify or attenuate harmful signals?

Related concepts in this collection 3

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
21 direct connections · 152 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

every step can pass its local check while the workflow still fails — SafeFlow's subtasks ChannelGuard's models and the Honest Quorum's validators