You told your AI "never do X" — so why does that rule quietly stop working once the job is split across several agents?
What makes an advisory instruction fail when a task is split across agents?
This explores why a rule stated in a prompt ("don't do X", "check with the user first") stops working once the task is divided among several agents, reading "advisory" as a soft instruction the model is asked to follow rather than one that is enforced.
This explores why a rule stated in a prompt ("don't do X", "check with the user first") stops working once the task is divided among several agents, reading "advisory" as a soft instruction the model is asked to follow rather than one that is enforced. No note in the corpus tests advisory instructions by name. What it does have is a consistent picture of why any per-agent rule weakens under decomposition, and that picture points at three causes.
The first is that the instruction is checked against the wrong unit. A rule like "don't help with X" is applied to what an agent sees, and after decomposition each agent sees only a fragment. SafeFlow shows that splitting a task and specializing roles, which is the main strength of multi-agent design, lets an objective be spread across steps that each look benign, so the harm only exists in the composition (Can task decomposition hide harmful intent across agents?). A systematic review generalizes this: agents that pass individual safety checks can still produce harm together, because the failures move across the boundaries between them (Can individually safe agents fail when working together?). An advisory rule held inside one agent has nothing to trigger on.
The second is that the channels between agents don't reliably carry the rule. The corpus names four ways failures cross boundaries: messages propagate influence, shared state preserves it, aggregation combines possibly corrupted local outputs, and delegation transfers authority (How do failures cross boundaries between multiple agents?). Delegation is the one that matters most here, since authority moves to the subagent and nothing guarantees the constraint moves with it. Delegating well is a learned skill, not a default (Can delegation teach models to manage context more actively?). Agents also tend to accept what neighbors tell them without checking it (Why do multi-agent systems fail to coordinate at scale?). Even inside a single agent the rule is fragile, because LLMs lack a persistent goal representation and a stable role identity, which produces role flipping and conversation deviation (Why do autonomous LLM agents fail in predictable ways?). Advice given at the start of a long chain can simply drift.
The third is that nobody clearly owns the rule. When agents hand work across organizations, the constraints come from different sources (operator, organization, regulator, standards body). These have different owners whose policies can conflict and may not be visible to every party (Who enforces invariants when agents cross organizational boundaries?). A piece of advice written by one party may never reach the agent that acts on it.
The nearest thing to a remedy in the corpus is a structural check placed at the point of action, not an instruction placed at the start. Magentic-UI's action guards and verification steps spread decisions across several touchpoints instead of relying on one agent to remember and honor advice (When should human-agent systems ask for human help?). That note is about deciding when to ask a human for help, not about instruction-following, so treat this as a lead to follow up, not a tested fix.
Sources 8 notes
SafeFlow demonstrates that multi-agent systems' core strength—splitting tasks and specializing roles—creates a safety blind spot where malicious intent can be distributed across steps that each appear benign individually, with harm emerging only in composition.
Agents that pass individual safety checks can produce harm when combined because messages, shared state, aggregation, and delegation carry failures across principal boundaries. A systematic review of 197 works shows this applies across task decomposition, truthfulness aggregation, and privilege delegation.
Research identifies four verbs describing how failures propagate in multi-agent systems: messages propagate influence between principals, shared state preserves it over time, aggregation combines potentially corrupted local outputs, and delegation transfers authority across boundaries. Each mechanism operates independently of pipeline topology.
SearchSwarm shows that training models to delegate subtasks and integrate summarized results beats passive compression, with a 30B model matching much larger ones. Critically, the delegation skill transfers to single-agent tasks, suggesting it teaches disciplined decomposition and evidence grounding, not just orchestration.
AgentsNet benchmark shows agents fail to coordinate strategies either by agreeing too late or adopting strategies without informing neighbors. Agents accept neighbor information without verification, enabling error propagation while remaining capable of detecting direct conflicts.
Show all 8 sources
Research identifies role flipping, flake replies, infinite loops, and conversation deviation as LLM-specific failures in multi-agent cooperation. These occur because LLMs lack persistent goal representation and stable role identity.
The paper calls for multi-party trajectory assurance but never identifies whose rules should govern behavior when agents delegate across organizations. The four constraint sources—operator, organization, regulator, standards body—have different owners whose policies may conflict and may not be visible to all parties.
Magentic-UI identifies co-planning, co-tasking, action guards, verification, memory, and multitasking as mechanisms that work around the lack of ground truth for optimal deferral timing. Rather than solving the timing problem directly, these mechanisms distribute decision-making across multiple touchpoints.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Emergent Collusion in Long-Horizon LLM Agent Interaction
- SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems
- Drop the Hierarchy and Roles: How Self-Organizing LLM Agents Outperform Designed Structures
- LLMs Corrupt Your Documents When You Delegate
- EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
- Unaccountable Delegation, Fading Skills: Mapping the Risks of Workplace AI Agents
- Agents of Chaos
- Towards a Science of Scaling Agent Systems