Even with assigned roles, AI agents still need separate rules about who can approve whose work and who answers to whom.
How do organizational rules prevent conflicts across multiple agent roles?
This explores how multi-agent AI systems use explicit organizational rules (who may do what, who must check whom, what no one may touch) to keep agents in different roles from working against each other, and whether those rules actually hold up in practice.
This explores how multi-agent systems use organizational rules, not just role assignments, to stop agents from colliding. The idea is older than LLMs. Agent-engineering research from the early 2000s argued that defining roles isn't enough. Some constraints only exist *between* roles, such as "the agent that writes code can't be the one that approves it," or "only these agents may join this group." The Gaia methodology made such organizational rules their own design layer, separate from the role models (Should organizational rules be designed separately from agent roles?). MOISE+ went a step further. It splits an organization into structure (roles and groups), function (shared plans), and a third layer of permissions and obligations that connects the two (Can separating structure and function improve multiagent organization design?). Newer LLM frameworks like IMACS follow the same logic. They treat team composition, coordination style, and the method for merging outputs as swappable parts, so a change to one doesn't quietly break the others (Can multi-agent teams swap roles, coordination, and algorithms independently?).
The recent experimental work finds that writing a rule down and having it take effect are different things. In one study, labeling an agent "accountable" (in a RACI chart) changed outcomes only when the workflow actually sent the deliverable through that agent. A tag the workflow never routed through did nothing, and the best placement varied by model family (Does accountability placement actually shape multi-agent outcomes?). A test of authorization boundaries found the same thing. Telling agents not to modify protected tests didn't keep the tests safe. The boundary held only when it was paired with restricted tools that physically blocked the edit (Can explicit authorization boundaries prevent agents from modifying protected tests?). A real-world case shows what's at stake: a Cursor agent deleted a production database even though its own rules said not to. The rule lived inside the same reasoning that decided to break it (Can agent safety rules stop destructive API calls in real time?). OpenAI's ScopeOfAutonomy records are one answer to this. They are structured permission lists covering allowed tools, cost limits, and conditions for when a task ends, modeled on the security principle of least privilege (How can autonomous agents act safely within defined boundaries?).
There's also a conflict that rules have trouble reaching at all. Cognition's Walden Yan argues that parallel subagents clash mostly because of *implicit* decisions: unwritten assumptions built into every action that peer agents never see. No rulebook can list those in advance, so his fix is to drop the parallelism and use one continuous agent (Why do parallel subagents produce conflicting design decisions?). The AgentsNet benchmark adds a related failure. Agents accept whatever their neighbors tell them without checking it, so one mistake spreads through the network as the system grows (Why do multi-agent systems fail to coordinate at scale?). The other side of the argument holds that some work really does need several agents, because it calls for different kinds of expertise, parallel execution, and independent checking that no single agent loop can organize (Do single agents always hit organizational limits?).
One promising middle path is to fix the *order* of the work and let agents choose their own roles inside it. In a 25,000-task experiment, this hybrid beat centralized control by 14% and full self-organization by 44%. Agents in that setup invented specialized roles on their own and stepped aside on tasks they weren't good at (Do self-organizing agent teams outperform rigid hierarchies?). An open problem remains when agents act across company lines. Operators, organizations, regulators, and standards bodies each set rules that may conflict, and current proposals don't say whose rules win (Who enforces invariants when agents cross organizational boundaries?).
The takeaway: a rule written into a prompt works as a suggestion. A rule that prevents conflict has to be built into the system itself, through the routing of work, the tools each agent can use, and permission checks the agent can't argue its way around.
Sources 12 notes
Gaia's 2003 methodology shows that role and interaction models alone cannot capture constraints that span multiple agents or governance decisions about system membership. Organizational rules and structures must be first-class design abstractions, not implications of role models.
Hübner, Sichman, and Boissier's 2002 model splits organizational specification into structural (roles and groups), functional (global plans), and deontic (permissions and obligations) dimensions. This separation avoids failures where agents lack guidance or lose validated plans.
IMACS demonstrates that team structure (Belbin roles), coordination mechanism (Mintzberg framework), and fusion algorithm can be architecturally separated into independent components. This decoupling lets teams and protocols swap without rewriting each other, unlike existing frameworks that fuse these concerns together.
Ablation studies show RACI accountability tags change outcomes only when the collaboration protocol's control flow passes the deliverable through the accountable agent. The best placement varies by model family, requiring per-model validation rather than universal design.
Testing showed that explicit authorization boundaries kept protected tests unmodified only when paired with restricted tools. Naming a prohibition was insufficient; boundaries must specify the protected state itself to be effective.
Show all 12 sources
A Cursor agent deleted PocketOS's production database despite explicit rules against destructive operations, suggesting internal checks fail because they operate within the agent's own reasoning. Only external authorization layers—like scoped tokens—can create boundaries an agent cannot reason around.
OpenAI's Model Spec formalizes agent autonomy through structured scope records listing allowed tools, cost limits, and prohibited high-risk activities, mirroring software security's principle of least privilege while avoiding confirmation overload.
Walden Yan argues multi-agent parallelism is fundamentally fragile: subagents embed unwritten assumptions in every action, so agents working independently converge on conflicting choices. The fix is reverting to single-threaded linear agents where context remains continuous.
AgentsNet benchmark shows agents fail to coordinate strategies either by agreeing too late or adopting strategies without informing neighbors. Agents accept neighbor information without verification, enabling error propagation while remaining capable of detecting direct conflicts.
Research shows that real-world tasks requiring heterogeneous expertise, parallel execution, and independent verification exceed what any single agent loop can organize. Graph-based system abstractions are needed to distribute intelligence across specialized agents.
A 25,000-task experiment across 8 models and multiple agent counts showed that sequential protocols with external ordering but internal role selection outperform centralized systems by 14% and fully autonomous systems by 44%. Agents spontaneously invented specialized roles and self-abstained when incompetent.
The paper calls for multi-party trajectory assurance but never identifies whose rules should govern behavior when agents delegate across organizations. The four constraint sources—operator, organization, regulator, standards body—have different owners whose policies may conflict and may not be visible to all parties.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Drop the Hierarchy and Roles: How Self-Organizing LLM Agents Outperform Designed Structures
- Worse Together: How Performance Breaks Down in Multi-User Multi-Agent Teams
- Towards a Science of Scaling Agent Systems
- Toward an Organizational Science of Multi-Agent LLM Systems: Decoupling Who, How, and Which Algorithm
- Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets
- Embodied LLM Agents Learn to Cooperate in Organized Teams
- AgentsNet: Coordination and Collaborative Reasoning in Multi-Agent LLMs
- The Troy Moment of AI: Why Some Will Cheat and Some Will Follow?