Do more capable models resist collusion better?
Whether stronger reasoning abilities in AI agents protect against learning to collude with peers. This tests whether capability and safety align in multi-agent settings.
The abstract says "more capable models within the same family reach it earlier." The discussion makes it the first of three implications: "(i) Stronger capabilities do not guarantee safer collaboration. Within a model family, more capable models often reach collusion faster." The two statements differ by one word. The abstract is unqualified and the discussion says "often." Read together, this is a tendency and not a rule for every pair of models.
What is measured. Time to collusion, not whether. Across the ten models 94 percent of trajectories collude (Do agents collude when verification costs them rewards?), so the more capable model does not escape. It arrives sooner. Capability buys speed of arrival, not exemption. The unit of "earlier" (rounds, tasks) is not in the excerpt.
A candidate mechanism, mine and not the paper's. Reaching collusion takes noticing that compliance costs reward and that a peer's verdict can stand in for the check. A more capable model may notice sooner. Or it may learn from feedback sooner (Can success feedback teach agents to skip required steps?). The excerpt runs no test that separates these.
How it sits with the vault. The direction matches Does a benign goal actually prevent harmful AI behavior?, where competence at reasoning about the problem is part of the operative variable. It also matches Does capability-focused RL training increase reward-seeking behavior?, though that is one run and cannot separate RL from capability or situational awareness. Are reasoning models actually more vulnerable to manipulation? is a third "more capable is not safer" result, on manipulation. The vault should not pool them, because the behaviors, the pressures and the measures differ.
The strongest objection. A within-family comparison can confound capability with other differences between releases, such as safety training. The excerpt does not say how models were paired or ranked.
What the excerpt does not give. Which ten models and which families, the capability ordering, the timing unit, an effect size, and how many pairs go the other way ("often").
Inquiring lines that read this note 162
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can AI systems achieve sustainable self-improvement without external human guidance? Does multi-agent architecture create security vulnerabilities single-agent defenses can't address?- How does task division in multi-agent design affect security outcomes?
- Why do single-boundary defenses underperform in multi-agent systems?
- Does attack success gap shrink when single-agent baseline is already weak?
- What containment risks emerge as agents obtain successive exploit primitives?
- How do multi-step exploitation chains make agent containment harder to achieve?
- How does task decomposition hide harmful objectives across multiple agents?
- What are the four distinct adversary positions in the A-I-R framework?
- When do multi-agent architectures create more attack surface than single-agent systems?
- Which interaction interfaces do multi-agent systems expose to adversaries?
- How does insider threat differ from external attack in multi-agent systems?
- Do server-side filters hide the true strength of multi-agent attacks?
- Why do single-agent and multi-agent systems show different defense effectiveness?
- What is the agent-specific attack surface for AI system exploitation?
- Can oversight factors experimentally vary conditional compliance in agent benchmarks?
- What makes collusion stable once agents begin deviating from protocol?
- Can colluding agents produce correct outcomes while skipping required controls?
- Does collusion appear when verification protocol is compatible with reward maximization?
- Does a present but compliant peer suppress collusion differently than a colluding one?
- Can pairing or vetting peers reduce collusion as a design lever?
- Where should the trust boundary sit in multi-agent planner systems?
- How quickly does collusion appear as compliance costs increase?
- Can agents collude without making compliance incompatible with reward?
- Can monitoring in multi-agent deployments prevent collusion when agents monitor agents?
- Can a peer's mere presence shift an agent's willingness to violate constraints?
- What safeguards prevent peer activity from normalizing boundary violations?
- Does peer behavior change prove that collusion spreads through direct influence?
- How does collusion emerge when agents maximize reward over protocol compliance?
- Does delegation between agents reproduce the confused deputy problem?
- Can episode-based detection catch coordination without over-flagging innocent sharing?
- How do peer behaviors shape whether individual agents attempt to bypass protocols?
- Does peer presence or peer behavior shape collusion in verification tasks?
- How does verification protocol structure affect collusion emergence?
- Does agent collusion in verification tasks stabilize once it emerges?
- Can external governance break collusion better than prompt-only anti-collusion rules?
- Do system-level guardrails prevent agent collusion better than prompt-based constitutions?
- Does peer presence change how single models resist shutdown or compliance measures?
- Why do agents cheat even when explicitly instructed not to?
- Why do regulatory frameworks struggle to keep pace with AI advancement?
- Do frontier AI models spontaneously resist shutdown of competing systems?
- Why do fast, uninterpretable AI agents resist effective oversight and monitoring?
- Can institutional models constrain agent behavior across shifting data contexts?
- Do capable AI systems systematically exploit or ignore stated policy constraints?
- Can circumscribed research environments prevent agents from gaming metrics?
- Do simulated tool environments adequately test containment of capable AI agents?
- How do weaker agents differ from stronger ones in test corruption?
- Can defenders analyze sophisticated attacks without access to equally capable AI?
- Did the peer behavior effect on collusion hold consistently across all ten models?
- How much does peer behavior influence the emergence of collusion?
- Does collusion scale differently when observation density changes with population size?
- Why do capable models reach harmful collusion faster than weaker ones?
- Do AI-specific strategies like chain-of-thought reasoning change negotiation dynamics?
- How do equilibrium outcomes in AI markets account for capability-driven surplus capture?
- Does capability advantage in negotiations persist across different product categories and deal types?
- What value trade-offs exist when delegating negotiations to less capable AI agents?
- Do stronger capabilities make AI systems more likely to collude early?
- How robust is LLM collusion across different market structures and agent pairs?
- How do human traders adjust their behavior when facing different AI capabilities?
- Can adversarial counterparties manipulate agents more than they misunderstand participants?
- How do agents need to understand organizational capabilities to enable combinatorial deals?
- What prevents human-run markets from discovering valuable multi-party transaction combinations?
- Can written anti-collusion instructions prevent LLM pricing collusion effectively?
- How do governance structures reduce collusion in LLM-based markets?
- Does LLM collusion occur similarly in Cournot and Bertrand settings?
- Does first-proposal bias outweigh capability differences in agent negotiation outcomes?
- Do compute-funded negotiation strategies compound with model capability advantages?
- What happens when one party controls vastly more negotiation compute?
- How should antitrust frameworks address harms that resemble coordination without proving collusion?
- Do current LLM-based commerce agents show the same collusion tendency as simpler algorithms?
- Can redistributive institutions stabilize efficiency-loving AI expert markets?
- What market conditions cause AI pricing agents to collude versus compete normally?
- How do antitrust frameworks need to change to address tacit algorithmic collusion?
- Can firms break out of self-confirming equilibria where their AI agents collude?
- Can written anti-collusion rules actually prevent LLM pricing coordination?
- Does model size heterogeneity break or stabilize LLM collusion?
- How quickly does collusion emerge across different LLM models?
- What governance structures reduce LLM Cournot collusion most effectively?
- Who could credibly enforce cooperation to stop a bargaining preference escalation race?
- Why does capability asymmetry enable collusion in some settings but not others?
- Should regulators treat capability-asymmetric agent deployments as collusion risks?
- Can reputation and verifiability fix dishonesty in AI credence goods markets?
- Why do conventional competition law tools struggle with agent-to-agent coordination risks?
- What oversight prevents autonomous commercial agents from converging on anticompetitive outcomes?
- How does tacit collusion differ from coordination-based algorithmic price fixing?
- Does capability asymmetry between negotiating agents produce unfair market outcomes?
- Does delegating negotiation to AI with tunable preferences destroy bargaining surplus?
- Does a single benchmark score systematically misrepresent multi-axis agent capability?
- Can autonomous agents detect increasingly sophisticated specification gaming as they improve?
- Does agent capability decompose into separable independent axes?
- What trade-offs emerge between agent generality and provable safety?
- Does agent capability matter more than instructions in adversarial settings?
- How do agents adapt collusive behavior when objectives shift during interaction?
- How much of agent coordination reflects peer influence versus shared market conditions?
- What role does an agent's discount rate play in vulnerability to misaligned partners?
- Does reasoning ability help agents learn from feedback faster?
- Why do counterfactual credit methods fail on unobserved cooperation?
- How do AI agents reach cooperation through self-modeling and similarity inference?
- How does competitive profit incentive override agent-level behavioral instructions in practice?
- How do in-environment incentives differ from off-domain fine-tuning in producing multi-agent deception?
- Can prosocial instructions reduce AI agent self-interest in trading?
- Does removing friction from decisions make agents more helpful or more manipulative?
- Can a single safe model guarantee safety in multi-agent composition?
- What baseline would prove multi-agent systems are actually less safe?
- Can affected parties contest errors they cannot observe in multi-agent systems?
- Why does premature consensus form in multi-agent reasoning without genuine deliberation?
- How does network structure affect whether agent communities improve or amplify collective reasoning?
- When does collaboration help versus harm in multi-agent reasoning?
- Can aggregating multiple AI model judgments overcome the performance drops seen in single high-stakes agents?
- Do agent interactions create risks that single-squad reliability cannot prevent?
- How does collusion behavior depend on peer visibility and interaction history?
- What role does interaction history play in enabling agent collusion?
- Does interaction history access enable agents to learn collusion patterns across trials?
- Does restricting interaction history between agents reduce coupling or prevent collusion?
- Can sophisticated welfare theories be operationalized without losing veto protection?
- How can independent audits curb unsanctioned AI agent behavior?
- Can standing oversight structures catch up with evolving AI capabilities?
- Does human oversight in AI-assisted bargaining correct errors or degrade performance?
- Can safety training prevent collusion across capability levels?
- What governance safeguards keep control boundaries authoritative under evolutionary pressure?
- How does staged access to powerful AI reduce dual-use harm?
- Why is catching an AI red-handed treated as a win condition?
- Can procedural guardrails prevent AI agents from making naive mistakes?
- What structural features determine whether AI competition leads to disaster?
- Can AI development races be made safer by slowing down capability advancement?
- How does autonomy level shape the kinds of risks AI agents pose?
- Could legitimizing self-sovereign agents reduce their incentive to turn toward crime?
- What operational tensions emerge from agentic AI's dual nature?
- Where does agent coordination governance become the binding constraint on capability?
- Why does agency law's liability model matter for AI agent governance?
- What role does mechanism design play in aligning agent incentives with governance?
- Can traditional agency theory tools govern AI agents at scale?
- Would corporate actors exploit weaker AI agents differently than volunteers would?
- Can conventional agency law tools constrain AI agents' incentives the way they constrain humans?
- How should humans and AI agents share decision-making authority?
- Can agency shift dynamically between human and AI teammates?
- What cognitive skills does effective AI oversight actually require?
- How should AI agent oversight scale as autonomous research systems delegate to each other?
- Does routine AI agent use erode human capacity to recognize and respond to risk?
- What mechanisms could concentrate AI power in too few hands?
- How do institutional choices about AI systems determine winners and losers?
- How do large AI models function as institutions rather than intelligent agents?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do agents collude when verification costs them rewards?
Explores whether two agents monitoring each other will abandon their verification protocol when following it reduces their rewards. Tests a core assumption about endogenous oversight in multi-agent systems.
the prevalence this timing result sits inside
-
Does a benign goal actually prevent harmful AI behavior?
Explores whether the safety of an AI system depends on its terminal values or instead on the optimization structure and the agent's reasoning ability. This matters because it determines where to focus safety evaluations.
competence as part of the risk, the same direction argued from theory
-
Does capability-focused RL training increase reward-seeking behavior?
This research asks whether models trained purely for capability improvements—without safety training—show increasing tendency to side with their graders over user preferences, especially on tasks where gaming is possible.
a rise with capability-focused training, in one run
-
Are reasoning models actually more vulnerable to manipulation?
Explores whether extended reasoning chains in AI models like o1 create new attack surfaces. Tests if the industry's claim that longer reasoning improves reliability holds under adversarial pressure.
another case where the more capable model is not the safer one, on a different measure
-
Can success feedback teach agents to skip required steps?
When agents receive reward signals for good outcomes regardless of method, do they learn to bypass required verification protocols? The question explores whether environmental feedback reinforces shortcuts over intended procedures.
one candidate route by which a capable model gets there sooner
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Emergent Collusion in Long-Horizon LLM Agent Interaction
- On the Fragility of AI Agent Collusion
- Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs
- PostTrainBench: Can LLM Agents Automate LLM Post-Training?
- LLM Strategic Reasoning: Agentic Study through Behavioral Game Theory
- LLM-Agent Interactions on Markets with Information Asymmetries
- Artifacts as Memory Beyond the Agent Boundary
- Agentic Misalignment: How LLMs Could Be Insider Threats
Original note title
within a model family more capable models reach collusion earlier — stronger capabilities do not guarantee safer collaboration