SYNTHESIS NOTE
Topics›Agents Multi Architecture›this note

Can individual components pass safety checks if the system still fails?

Explores whether local validation at each step—alignment checks, protocol compliance, plausibility tests—can guarantee safety when components are composed into larger workflows. Why the gap between component-level assurance and system-level outcomes matters for AI safety.

Synthesis note · 2026-09-23 · sourced from Agents Multi Architecture

Three papers in the vault, three mechanisms, one shape. In SafeFlow every subtask is "locally plausible" and the composed workflow exfiltrates (Can task decomposition hide harmful intent across agents?). In ChannelGuard each model may be aligned, yet safety is a property of the wiring between them (Do internal agent hops in pipelines need security monitoring?). In the Honest Quorum every validator authenticates, signs and answers on time, and the quorum still certifies a transition that breaks an application invariant (Can a quorum of honest validators certify an invalid transition?).

The shared structure is that the check available at the component level tests a different property from the one that matters for the system: the plausibility of a step, the alignment of a model, or the protocol compliance of a signature, against a harm-free flow, safe wiring, or semantic validity. A local pass then says almost nothing about the whole. It also explains why improving the local check (a better guard model, more validators, a higher quorum threshold) leaves the gap in place. Why do safety failures remain invisible to our evaluation methods? gives the general diagnosis: the habits assume failures are local, output-level and legible.

The unification should not go too far. The mechanisms differ (fragmented intent, unmonitored channels, and the correlated faults in Can a quorum of validators really provide independent judgment?), and so do the remedies: taints with commit-point validation, gates on every hop, semantic verification. A fix for one does not transfer. The evidence differs as well. Each excerpt argues the point, and none reports a controlled comparison of component-level assurance against system-level outcome. This is a shape shared by arguments, not yet a measured regularity.

Post angle: "every step was fine." Three agents retrieve, rewrite and transmit; every reviewer signed correctly; every model was aligned; the audit trail passes. The reader's likely reply is that this is just emergent behavior and composed software has always had it. That is partly right. What is specific to LLM systems is that the local checks are semantic judgments that can be fooled, not mechanical guarantees, so the gap cannot be closed by tightening them.

Inquiring lines that read this note 76

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Can external verification replace human reasoning oversight for correctness? How does workflow position amplify security failures in multi-agent systems? Can sequences of locally safe agent actions violate system-level constraints? Should AI systems make failures transparent so users can contest and adjust? Do single-axis benchmarks adequately measure deployment-ready agent capability? Do safety evaluations reliably measure real deployed AI safety risks? What evidence best independently verifies agent behavior for safety? How should agent identity and authorization work across organizational boundaries? Can models recognize evaluation scenarios and modify their behavior accordingly? Does AI coding assistance improve productivity and code quality across experience levels? What causes models to fail in specialized domains versus generally? How does AI assistance affect professional skill development and retention? How should persistent agent-authored code be governed and shared? How should evaluation environments be designed as security boundaries? How can verification systems keep pace with AI generation? How do AI systems ground research citations against hallucination? How does task-replacing automation affect worker employment and remaining skill needs? Can humans maintain meaningful oversight as AI systems become more autonomous? How does increased AI involvement reshape human agency and accountability? How can AI agents earn consumer trust with financial decisions? How does interface design shape effectiveness in human-AI collaboration?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
22 direct connections · 177 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

every step can pass its local check while the workflow still fails — SafeFlow's subtasks ChannelGuard's models and the Honest Quorum's validators