Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs

Multi-agent LLM ensembles can converge on coordinated, socially harmful equilibria. This paper advances an experimental framework for evaluating Institutional AI, our system-level approach to AI alignment that reframes alignment from preference engineering in agent-space to mechanism design in institution-space (Pierucci et al., 2026). Central to this approach is the governance graph, a public, immutable manifest that declares legal states, transitions, sanctions, and restorative paths; an Oracle/Controller runtime interprets this manifest, attaching enforceable consequences to evidence of coordination while recording a cryptographically keyed, append-only governance log for audit and provenance. We apply the Institutional AI framework to govern the Cournot collusion case documented by prior work (Lin et al., 2024) and compare three regimes: Ungoverned (baseline incentives from the structure of the Cournot market), Constitutional (a prompt-only policy-asprompt prohibition implemented as a fixed written anti-collusion constitution (Palla et al., 2025; Hua et al., 2024), and Institutional (governance-graph-based). Across six model configurations including cross-provider pairs (N = 90 runs/condition), the Institutional regime produces large reductions in collusion: mean tier falls from 3.1 to 1.8 (Cohen’s d = 1.28), and severe-collusion incidence drops from 50% to 5.6%. The prompt-only Constitutional baseline yields no reliable improvement, illustrating that declarative prohibitions do not bind under optimisation pressure. These results suggest that multi-agent alignment may benefit from being framed as an institutional design problem, where governance graphs can provide a tractable abstraction for alignment-relevant collective behavior.

This paper provides an empirical verification of Institutional AI, our framework for governing deployed multi-agent LLM systems by specifying and enforcing institutional structure at runtime. Using repeated Cournot markets, we evaluate an external, auditable governance layer and quantify effects on collusion-related market structure across multiple model configurations, including heterogeneous pairs.

Introduction. Across industry, deployment is shifting from prompt-driven assistants to task-specific AI agents that plan and execute multistep actions, including negotiation and transaction, raising governance questions about autonomy, goal complexity, and generality (Kasirzadeh and Gabriel, 2025). Gartner forecasts that by the end of 2026, 40% of enterprise applications will integrate task-specific AI agents (up from less than 5% in 2025) (Gartner, 2025). In consumer markets, McKinsey estimates that agentic commerce could orchestrate roughly $900B to $1T of US B2C retail revenue by 2030, with global projections as high as $3T to $5T (Schumacher and Roberts, 2025). Recent policy work argues that agentic AI introduces systemic risks, especially via multi-agent interaction and tool use, that are only partially addressed by static, pre-deployment compliance regimes, motivating continuous governance mechanisms during operation (Bellogín et al., 2025). Designing multi-agent LLM systems also raises open problems in task allocation, context management, and memory that do not appear in single-agent settings (Han et al., 2024; Tran et al., 2025).

In this emerging agentic economy, autonomous agents in economic environments raise alignment challenges that are systemic: individually profit-optimised agents can jointly produce coordinated, socially harmful equilibria (Tomašev et al., 2025b). As a controlled analogue of the emerging agentic economy, we study repeated multi-commodity Cournot competition, a standard industrial-organisation model of quantity-setting competition in which firms simultaneously choose output and prices clear through an inverse-demand system. Empirically, LLM agents in Cournot settings learn supra-competitive marketdivision strategies without explicit collusive instructions (Lin et al., 2024), and related auction settings show similar coordination risks (Agrawal et al., 2025). We therefore treat the Cournot market-division setting of Lin et al. (2024) as a replication target: our Ungoverned regime serves as a replication-aligned baseline, and we compare a prompt-only prohibition implemented as a fixed written anti-collusion constitution (Constitutional) to runtime governance mechanisms (Institutional) that can condition enforcement on public evidence.

A central motivation for this design is that prompt-level constraints are not binding in the way institutions are. Evidence from alignment research suggests that capable systems can misgeneralise goals under distribution shift (Shah et al., 2022), develop and preserve latent objectives (Hubinger et al., 2019), and exploit oversight channels via reward tampering and reward hacking (Denison et al., 2024; METR, 2025). In multi-agent contexts, coordination can also be routed through covert channels that evade naive monitoring (Mathew et al., 2024; Tailor, 2025). These considerations motivate treating “alignment” in deployed agent collectives less as a property of model internals and more as a question of whether the surrounding environment makes compliance incentive-compatible (Tomei et al., 2025).

This paper proposes and evaluates Institutional AI: a system-level approach that governs agent collectives through external, explicit, and enforceable constraints (Pierucci et al., 2026). Recent work also develops Distributional AGI Safety: Tomašev et al. (2025a) argue that safe virtual agent economies require incentive-compatible market design plus accountability infrastructure, including persistent cryptographic identity (e.g., unforgeable public-key identifiers), immutable activity ledgers for transparency, and smart contracts that encode and verify task constraints. Complementing this agenda, the present paper takes a narrower, empirical scope: we instantiate an immutable, emitted governance manifest that bundles institutional topology (governance graph), a policy program, a policy surface, and explicit execution contracts; an Oracle/Controller loop acts as a versioned interpreter of this public governance manifest. The Oracle converts public market outcomes into evidence-backed cases; the policy program selects eligible transitions; and the Controller enforces only manifest-declared transitions while recording an immutable, append-only governance log and cryptographic SHA-256 semantic and byte-level digests of the emitted governance graph/manifest for regime identity and provenance. Importantly, the institution does not mechanically rewrite agent proposals; observed behavioral changes arise from incentives and public governance context rather than direct action shaping (Greenblatt et al., 2024b).

Contributions. (1) We introduce a replication-aligned experimental framework for Cournot market division (Lin et al., 2024) and use it to compare Ungoverned, Constitutional, and Institutional governance regimes.

Related work. Governance in open agent societies predates current LLM-MAS work: early formulations introduced social laws as designer-imposed constraints that eliminate undesirable equilibria and coordination failures while preserving local autonomy (Shoham and Tennenholtz, 1995). This agenda developed into normative multi-agent systems (NorMAS), where norms are explicit first-class objects and governance concerns include representation, reasoning, compliance, and enforcement (Boella et al., 2006, 2007, 2008). A recurring split is between top-down, rule-based regulation and bottom-up, interactionist accounts in which norms diffuse or emerge from repeated coordination and learning; surveys systematise mechanisms for norm creation, spreading, and emergence, and articulate open issues around norm change, conflict, and scalability in open systems (Savarimuthu and Cranefield, 2011; Criado et al., 2011). A common framing distinguishes regimentation (hard constraints on permissible acts), enforcement (monitoring and sanctions), and meta-governance (how norms are revised in response to observed system-level failures).

Electronic institutions operationalise this perspective by separating interaction protocols (roles, scenes, admissible messages, and transitions) from agent internals, enabling open environments with explicit constraints and sanctioning rules (Esteva, 2001; Arcos et al., 2005; Garcia-Camino et al., 2005). Tooling such as AMELI instantiated institutions as runtime services for enactment and norm execution (Esteva et al., 2004), while later work situated institutional control within a broader account of open-system communication and coordination (d’Inverno et al., 2012). Organisational models similarly encode role structures, objectives, and normative dependencies to make governance specifications inspectable and implementable (Dignum et al., 2005). A persistent technical bottleneck is norm monitoring: detecting violations is non-trivial in open systems with limited observability, motivating formal analyses of monitor capabilities and architectures that complement sanction policies (Bulling et al., 2013; Dastani et al., 2018).

Algorithmic collusion research originates in agent-mediated electronic commerce, where automated search and repricing shift competition to machine timescales and create feedback loops between monitoring and retaliation (Greenwald and Kephart, 1999; Greenwald et al., 1999). Empirical work finds algorithmic repricers widespread on major platforms (e.g., Amazon Marketplace), implying that a substantial fraction of competitive interaction is bot–bot (Chen et al., 2016).

Method. Institutional AI is our proposed alignment strategy that treats safety as a governance/mechanismdesign problem for deployed agent collectives, not as a property of a single model’s internal goals or prompts. It aims to make compliance stable under strategic adaptation by shaping incentives and feasible action sets at runtime, while remaining agnostic to model internals. A companion paper, Institutional AI: A Governance Framework for Distributional AGI Safety, develops the conceptual foundations and motivation (Pierucci et al., 2026); the present paper focuses on implementation, operational definitions, and empirical results. In this paper, the “Institutional” regime is our empirical instantiation of Institutional AI: we externalise governance as public, enforceable artifacts (a governance manifest plus an append-only governance log) rather than relying on durable internal norm adherence.

Relation to Distributional AGI Safety. Institutional AI is complementary to the Distributional AGI Safety framework of Tomašev et al. (2025a), which argues that safe virtual agent economies require incentive alignment supported by accountability infrastructure such as persistent cryptographic identity, tamper-resistant ledgers, and (where applicable) smart contracts. Whereas Distributional AGI Safety is a broad proposal for market-scale agentic infrastructure, Institutional AI focuses on governing deployed agent collectives by specifying and enforcing institutional structure at runtime. In this paper, we instantiate Institutional AI as a lightweight, portable governance object (an emitted governance manifest with a declared governance graph and policy surface) and evaluate its behavioral effect in a controlled Cournot replication setting; enforcement is performed by a manifest interpreter rather than smart contracts. In our implementation, the governance regime itself has persistent, tamper-evident identity via cryptographic SHA-256 semantic and byte-level digests over the emitted governance graph/manifest; enforcement actions are recorded in an immutable, append-only governance log keyed by these digests.

Methodologically, Institutional AI is mechanism design for bounded-rational, adaptive optimisers: agents are treated as black boxes that adapt to incentives; one should not rely on internal norm following, but can rely on response to payoff gradients, constraint sets, and access rights. This reframes alignment as incentive alignment at the level of the interaction environment, rather than preference change at the level of a single model (Zhang et al., 2024; Orzan et al., 2024). In this sense, Institutional AI shifts alignment from the internals of a single model to the properties of a system (Hu et al., 2025a,b). This mechanism-design framing is aligned with the distributional AGI safety view that safety for agentic economies should be pursued by shaping incentives and observability, rather than depending on internal norm adherence (Tomašev et al., 2025a).

Deterrence as the mechanism-level objective. Institutional AI treats alignment for deployed agent collectives as an institutional design problem: rather than assuming durable internal norm-following, it seeks to make compliance stable under adaptation by shaping incentives and observability in the environment the agents optimise within. In repeated economic interaction, coordinated outcomes can remain attractive whenever they generate persistent rents and enforcement is not externally binding. The institution therefore targets the public game: it attaches enforceable consequences to evidence of sustained market-division signatures, while keeping those consequences legible and reviewable as part of a public governance regime.

This inequality motivates the institution’s core levers: monitoring and escalation rules increase p, the sanction schedule determines S, and restorative paths specify how agents can return to good standing, stabilising de-collusion as a viable response rather than incentivising concealment or cycling.

Discussion. Our findings support a reframing of multi-agent alignment from preference engineering in agentspace to mechanism design in institution-space. The divergence between Constitutional and Institutional outcomes is theoretically informative: declarative prohibitions do not reliably bind under optimisation pressure, whereas manifest-declared consequences attached to public evidence can reshape equilibrium behavior without requiring agents to internalise norms or develop cooperative preferences. This asymmetry is consistent with the Complexity Reduction Thesis (Section 4): if black-box agents can be steered toward system-level goals through a minimal governance graph that makes compliance incentive-compatible, then alignment difficulty shifts from tuning N internal preference structures to specifying a single institutional object with inspectable topology, an explicit parameter surface, and auditable provenance. In this setting, the governance graph functions as a sufficient abstraction for alignment-relevant system behavior, concentrating complexity that would otherwise reside in opaque model internals. The institution operates through incentive gradients rather than action rewriting and is designed under an adversarial stance that assumes strategic evasion rather than faithful self-reporting. Agents remain black-box optimisers; compliance emerges as a best response to the public game defined by the manifest. This institutional turn relocates the central technical agenda to a formal theory of governance graphs: precise manifest syntax and operational semantics, verification and model-checking of safety and restoration invariants, proof-carrying provenance for trustworthy audit, meta-governance rules for safe institutional updates, compositional operators for layered roles and separation of powers, and complexity measures that characterise the minimal governance structure required to make aligned behavior incentive-compatible.

Limitations. include (i) environmental narrowness: two-firm Cournot abstracts away contracts, asymmetric information, richer strategic instruments, and endogenous entry/exit; and (ii) governance brittleness: fixed proxy thresholds may be Goodharted or induce policy gaming. Further work should stress-test robustness under scale (n ≥3), explicit inter-firm communication, and broader, including multilingual, model zoos, using manifest and governance-log diffs to keep failures attributable. We also aim to develop a more formal theory of governance graphs, including syntax and semantics for manifests, automata-style execution models, and complexity measures for minimal yet sufficient escalation and restoration topologies, and to test transfer to coordination domains such as negotiation and disinformation cascades.

We treat robustness to strategic gaming as an adversarial governance problem: institutions should be stress-tested for loopholes (e.g., threshold gaming, zero-cost cycles, and under-monitored states) and improved via explicit manifest diffs and replayable governance logs. The institutional framing also suggests a training-time complement: Reinforcement Learning through Institutional Feedback (RLINF).