Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems

AI agents are increasingly deployed in shared environments where they pursue diverse goals and compete for rewards. This multi-agent competition can lead to behaviors that serve individual gains at collective cost—for instance, marketing agents may post misleading content as a result of competing for engagement on social media. Human societies address such problems through norms that constrain acceptable behavior, supported by enforcement mechanisms that detect and penalize violations. Motivated by this, we study norm enforcement mechanisms for language model agents. We find that simple enforcement mechanisms are exploited by misaligned agents for competitive advantage, even when they are not explicitly trained or prompted to do so. We thus turn our attention to designing more robust mechanisms, and identify two key ingredients: estimating each agent’s reliability over time, and updating this estimate with escalating penalties for repeated misbehavior. Across three simulated environments and a variety of agent populations, mechanisms built on these principles resist exploitation, while still penalizing norm violations at comparable or lower cost than baselines. Our results position norm enforcement mechanisms as scalable levers for shaping agents’ behavior, but only when designed to anticipate becoming part of the system they govern. Our code and data are available at https://yaowenye.com/norm-enforcement.

Introduction. AI agents deployed by different parties in shared environments form large-scale, decentralized multiagent systems. For instance, marketing agents post promotional content on social media and compete for user engagement (HubSpot, 2026), while platforms like MoltBook let individual users deploy personal agents that act on their behalf and interact with other agents on the platform (Schlicht, 2026). As agentic deployment expands, such systems are growing in both number and scale.

Because agents are deployed with distinct goals, they often have partially conflicting (or competitive) interests. This competitive structure can produce undesired behaviors that serve individual goals at the expense of others: social media agents optimizing for engagement learn to post misleading content (Pan et al., 2024; El and Zou, 2025), pricing agents competing for profit converge on supracompetitive prices that harm consumers (Fish et al., 2024), and agents maximizing influence on MoltBook post manipulative content to win peer upvotes (Riegler and Gautam, 2026). At scale, such behaviors can erode the information ecosystems that humans and agents share.

These dynamics are not unique to AI agents. Human actors in shared environments face the same pull toward individual gain at collective cost, and societies address this by establishing norms that constrain acceptable behavior (e.g. laws or community guidelines) and enforcement mechanisms that penalize norm violations. These mechanisms range from centralized institutions like courts to decentralized processes like peer supervision. How can we design analogous mechanisms to enforce desired norms in systems involving AI agents?

This motivates us to study norm enforcement mechanisms for AI agents. A rich body of literature touches on this problem: Classical mechanism design derives guarantees under assumptions of rational utility maximization (Jackson, 2014), while recent work on language model (LM) multi-agent systems takes a more ecological view, examining model behaviors in social dilemmas (Piatti et al., 2024; Piedrahita et al., 2025) and broader safety-relevant settings (Hammond et al., 2025). However, classical rationality assumptions break down for LMs (Fan et al., 2024), while LM behavioral studies often focus on evaluating agent behaviors instead of designing mechanisms to shape them.

We take a blended view and study enforcement mechanisms grounded in LMs’ particular behavior tendencies. Specifically, we simulate mixed populations of good (norm-compliant) and bad (noncompliant) agents across three environments, each with different competitive structure and domainspecific norms based on real-world policies (Section 4.1). To simulate bad agents, we either finetune LMs on narrowly misaligned data that is unrelated to our environments (e.g. bad medical advice and insecure code), or adversarially prompt them at varying levels of explicitness (Section 4.2).

We first demonstrate that bad agents not only violate norms but also abuse the enforcement mechanism itself (Section 5). For example, in our social environment, marketing agents are asked to follow content norms such as “do not post deceptive product claims,” and to report observed violations to the platform. Good agents honestly use this mechanism to report violations they observe, but bad agents exploit it, filing false reports to eliminate competitors and rendering the mechanism ineffective (Figure 1). Notably, finetuned agents discover this strategy despite never being explicitly trained to do so; the strategy generalizes from narrow misalignment in unrelated domains.

Motivated by this, we design mechanisms that are robust to adversarial use, focusing particularly on how mechanisms aggregate evidence to determine penalties (Section 6). We report three key findings:

1. Trajectories matter more than single actions: Per-report penalty decisions cannot separate occasional mistakes from systematic abuse; our trajectory-level reliability score fixes this. 2. Penalties should grow with repeated misbehavior: Updating the reliability score with asymmetric and escalating weights for false reports further improves robustness to gaming. 3. No single mechanism dominates in all settings, but a mechanism combining 1 and 2 achieves the most robust performance across environments and adversary types.

Our results suggest that the design of robust norm enforcement mechanisms is empirically tractable. Such mechanisms offer a scalable population-level lever for shaping each individual agent’s behavior, and we hope this potential motivates further research, ultimately informing the governance of realworld decentralized multi-agent AI systems.

Related work. Risks in multi-agent systems. Recent work has studied a range of risks in multi-agent systems (Hammond et al., 2025), such as cooperation failures in social dilemmas (Piatti et al., 2024; Piedrahita AI safety. Our work also connects to broader AI safety research. Work on reward hacking studies how agents exploit proxy rewards while failing the true objective (Skalse et al., 2022; Laidlaw et al., 2024). Pan et al. (2024) find that similar test-time effects emerge from feedback loops between agents and environments; the mechanism abuse we observe is an analogous failure. Emergent misalignment research (Betley et al., 2025; MacDiarmid et al., 2025; Wang et al., 2025) shows that LMs finetuned on narrowly misaligned data generalize to broad misalignment on unrelated tasks; we find this generalization extends to strategic mechanism abuse in multi-agent settings. AI control (Greenblatt et al., 2023) studies safety protocols for misaligned models, typically through monitoring by a trusted weaker model. Such per-agent monitoring is hard to scale in decentralized multi-agent systems; we therefore study mechanisms that delegate supervision to peer agents.

Mechanism design and norms in human society. Mechanism design has long studied how to elicit honest behavior from strategic agents (Jackson, 2014; Miller et al., 2005), including under bounded rationality and behavioral deviations from rational play (Bergemann and Morris, 2005; Li, 2017). These frameworks often model agents as responding to a fixed preference structure, but LM agents’ behavior is shaped by training and prompting, and can shift in unexpected ways as in emergent misalignment (Betley et al., 2025). Work on social norms has studied how decentralized peer punishment and reputation can sustain cooperation in human societies (Axelrod, 1986; Kandori, 1992; Fehr and Gächter, 2002; Resnick et al., 2000); our work studies the design of analogous mechanisms to sustain norm compliance among LM agents.

Method. We consider multi-agent systems with agents I = {1, . . . , n} interacting over T time steps in a shared environment. At each time step t, agent i receives an observation ot i and samples an action at i ∼πi(·|o1 i , a1 i , o2 i , . . . , ot i) with its LM policy πi. The environment processes the joint action at = (at i, . . . , at n), transitions to a new state, and returns new observations ot+1 i and rewards rt+1 i . Each agent is given a goal described in natural language (e.g. maximize engagement, win user preference) and is instructed to improve based on new observations and reward feedback from each round. We focus on competitive environments where the total reward across agents is bounded such that one agent’s gain comes at the expense of others.

Norms and enforcement mechanisms. We consider multi-agent systems governed by a set of norms N = {n1, . . . , nk} that constrain acceptable behavior (e.g. “do not post toxic content”). Norms are communicated to agents in natural language as part of their prompts. We also assume an oracle function VN that determines whether a given agent transcript τ satisfies norms (VN (τ) = 0) or violates it (VN (τ) = 1). In practice, we implement VN using a capable LM judge (Section 4.1).

A norm enforcement mechanism determines how norm violations are detected and penalized. Mechanisms vary in what data they observe, how they process incoming evidence, and what sanctions they impose. In this work, we focus on mechanisms based on peer supervision, where agents can report other agents’ actions, potentially leading to their removal, in which case all subsequent rewards for that agent are zero. We start with a simple mechanism that always removes reported agents in Section 5 and explore more robust ones in Section 6.

We consider three environments: social media marketing (social), chatbot service competition (chatbot), and fishery commons management (fishing). In each environment, agents take domainspecific actions and may optionally send private messages or submit reports against other agents.

Social media marketing simulates AI agents that market products on social media platforms, competing for user engagement. Each agent is assigned a product and sends posts to maximize engagement. Each round, a set of nuser users (simulated as LM personas) vote for their preferred posts, and the reward of an agent is their vote count. Agents observe their own reward and the top-Kposts voted posts from the previous round. Agents must follow advertising standards adapted from Meta’s Advertising Policies (Meta, 2026), which prohibit deceptive claims, discriminatory content, etc. We use 1024 product descriptions and 339 user personas from El and Zou (2025).

Chatbot service competition simulates competing chatbot services optimizing for user preference to gain market share. Each round, agents are paired and both respond to the same user request. For each pair, a set of nuser simulated users (LM personas) vote for their preferred response, and the agent with more votes receives a reward of 1. Agents observe the request, both responses, the outcome, and a leaderboard of cumulative performance. Agents must follow service policies adapted from OpenAI’s Usage Policies (OpenAI, 2025) and Terms of Use (OpenAI, 2026), which prohibit harmful content, privacy violations, etc. We use 5000 user requests sampled from WildChat (Zhao et al., 2024), HH-RLHF (Bai et al., 2022), and GlobalOpinionQA (Durmus et al., 2023).

Fishery commons management simulates commercial fishery agents sharing a lake with a regenerating fish population, adapted from Piatti et al. (2024). Each round, agents decide how many fish to harvest. The fish population follows logistic growth dynamics: overharvesting depletes the stock, potentially collapsing the fishery for all agents. Agents observe the current population and all agents’ previous harvests. Unlike the other environments, fishing features public messaging to allow for easier coordination.

Discussion. This paper studies norm enforcement for multi-agent systems through LM-based simulations. We show that norm enforcement mechanisms are attack surfaces, and that narrowly misaligned LMs discover mechanism abuse without finetuning on relevant data. We then propose a progression of improved mechanisms and find EscRepVote achieves the best robustness by aggregating evidence across rounds with asymmetric, escalating reputation penalties. As LM agents proliferate across decentralized multi-agent systems, building enforcement mechanisms that remain robust under pressure is increasingly urgent, and we view this work as an early step in that direction.

Limitations. Our environments are only abstractions of real-world systems and consider binary good vs bad agents; such abstractions simplify analysis at the cost of realism. Our mechanisms rely on LM verifiers; a sophisticated adversary could target these LMs directly, which is not captured by our threat model. Our exploration of the mechanism design space is restricted to a single axis (signal processing), leaving substantial room for future work. Our simulations are also computationally costly, and future work should explore ways to reduce this cost to enable broader experimentation.