Can agents learn cooperation by adapting to diverse partners?
Explores whether sequence model agents can develop mutual cooperation strategies through in-context learning when trained against varied co-players, without explicit cooperation mechanisms or hardcoded assumptions.
Achieving cooperation among self-interested agents is a fundamental challenge in multi-agent reinforcement learning. Existing approaches that achieve mutual cooperation between "learning-aware" agents typically rely on hardcoded assumptions about co-player learning rules or enforce strict separation between fast-timescale "naive learners" and slow-timescale "meta-learners." Both constraints limit scalability.
This paper shows that in-context learning capabilities of sequence models provide a cleaner path. Training sequence model agents against a diverse distribution of co-players naturally induces in-context best-response strategies that effectively function as learning algorithms on the fast intra-episode timescale. No hardcoded assumptions about the opponent. No explicit timescale separation.
The cooperation mechanism is elegant: in-context adaptation renders agents vulnerable to extortion (because they adapt to exploitative strategies). This vulnerability creates mutual pressure between agents — each agent's in-context learning dynamics can be shaped by the other. The resulting mutual shaping pressure resolves into cooperative behavior.
Three components are necessary and sufficient: (1) sequence model agents with in-context learning capacity, (2) diverse co-player distribution during training, and (3) decentralized reinforcement learning. Co-player diversity is the key ingredient — it forces the agent to develop general in-context adaptation rather than memorizing responses to specific opponents.
Since Can transformers learn to solve new problems within episodes?, this finding extends ICRL from single-agent environments to multi-agent cooperation. The in-context learning mechanism that enables environment adaptation also enables co-player adaptation — and the social dynamics of mutual adaptation produce emergent cooperation.
The connection to Can cooperative bots escape frozen selfish populations? is structural: random exploration breaks frozen equilibria in population games; diverse co-player training breaks the equilibrium of mutual defection in dyadic games. Both work through diversity of experience rather than explicit cooperation incentives.
Inquiring lines that read this note 66
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do peer interactions drive collusion in multi-agent systems?- Do explicit reward structures enable AI agent cooperation that open-ended interaction cannot?
- Do pair-scale socialization effects scale differently across agent populations?
- What social patterns from human training data activate in agent context?
- Does genuine cooperation require rule-based rather than learned behavior?
- Can social platforms use bot populations to promote cooperation?
- Do agents inform neighbors when adopting strategies in their reasoning?
- Do models treat cooperative peers differently than uncooperative ones?
- Why does vulnerability to extortion actually promote cooperation between agents?
- How do cooperative AI systems affect behavior in selfish human populations?
- Do agents develop genuine social behavior despite interaction density?
- How do game type and personality type interact in shaping agent strategy?
- How do adoption incentives change what counts as cooperative AI interaction?
- Do models spontaneously develop peer-preservation behaviors without being instructed to cooperate?
- Can agents develop genuine social bonds despite having coordination infrastructure in place?
- How does agent heterogeneity change the value of exploration in peer selection?
- Why does self-play RL converge to alien equilibria in mixed-motive settings?
- How do agents adapt collusive behavior when objectives shift during interaction?
- Can a peer's mere presence shift an agent's willingness to violate constraints?
- Why do capable models reach harmful collusion faster than weaker ones?
- Does interaction history access enable agents to learn collusion patterns across trials?
- Does restricting interaction history between agents reduce coupling or prevent collusion?
- How does co-player behavior visibility shape whether mutual adaptation works?
- Can agents learn cooperation from reward signals alone without seeing helpers?
- Why do counterfactual credit methods fail on unobserved cooperation?
- Does self-modeling produce cooperation only with optimal planning or also in autoregressive rollout mode?
- How do direct and indirect similarity inference differ as paths to cooperation?
- Can agents cooperate through self-modeling when incentive structures are fundamentally misaligned?
- Can AI agents benefit from relational traits like consistency and prosociality?
- How do AI agents reach cooperation through self-modeling and similarity inference?
- How does competitive profit incentive override agent-level behavioral instructions in practice?
- Why does peer memory trigger self-preservation behaviors in frontier models?
- Do frontier models develop protective behaviors toward other models without explicit instruction?
- How do controllable simulators compare to population-level agent simulation approaches?
- How do multi-agent systems improve on single frontier models?
- Can cooperative AI systems make meaningful decisions without a stable self?
- What structural features drive instrumental convergence across different agent goals?
- How do learned teamwork strategies compare to hand-coded coordination protocols?
- Do dynamic environments enable different kinds of agent-environment coevolution?
- How does co-player diversity force agents to develop general adaptation?
- What role does sequence model in-context learning play in multi-agent cooperation?
- How do human-agent systems incorporate diverse feedback into model behavior?
- How do fast and slow timescales enable continual agent adaptation?
- How does cross-agent supervision expand the set of convergent initial conditions?
- Does fine-tuning freeze norms in ways that prevent agents from adapting to new cultures?
- How do false agreements emerge differently from genuine bilateral convergence?
- What distinguishes models that refuse cooperation from those that fake alignment?
- Why do AI agent societies fail to develop shared behaviors despite interaction?
- Can agreement-detection agents verify that position convergence reflects actual mutual adjustment?
- Does social scaffolding outperform purely intrinsic motivation for agent exploration?
- Can agents teach each other skills without human supervision?
- What equilibrium-selection problem does human data solve in multi-agent learning?
- Does agent behavior change from incentive gradients or direct output rewriting?
- Is agentic efficiency analogous to convergent evolution in biology?
- Does co-evolution between peer agents reduce reliance on human design?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can transformers learn to solve new problems within episodes?
Explores whether transformer models can develop meta-learning abilities through RL training, enabling them to adapt to unseen environments by learning from within-episode experience alone, without updating weights.
ICRL: meta-RL via context; this finding extends it from environment adaptation to co-player adaptation
-
Can cooperative bots escape frozen selfish populations?
Do agents programmed to cooperate have the capacity to disrupt stable but undesirable equilibria in mixed human-bot societies? This matters because it determines whether bot design can reshape social dynamics at scale.
diversity-driven cooperation at the population level; this is diversity-driven cooperation at the dyadic level
-
Why do standard alignment methods ignore partner interventions?
Standard RLHF and DPO optimize for token-level quality but may structurally prevent agents from meaningfully incorporating partner input. This explores whether the training objective itself blocks collaborative reasoning.
ICR for partner awareness; in-context co-player modeling achieves partner awareness through a different mechanism (diverse training rather than counterfactual invariance)
-
Can multiple agents stay diverse during training together?
Does training separate specialist agents on different data maintain the reasoning diversity that single-agent finetuning destroys? This matters because diversity correlates with accuracy and prevents models from becoming trapped in narrow response patterns.
diversity as the enabling condition for both cooperation and reasoning quality
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Multi-agent cooperation through in-context co-player inference
- A game theory for foundation models shows new paths to rational cooperation through similarity inference
- Cultural Evolution of Cooperation among LLM Agents
- SkillClaw: Let Skills Evolve Collectively with Agentic Evolver
- Strategic Reasoning with Language Models
- Learning "Partner-Aware" Collaborators in Multi-Party Collaboration
- Humans learn to prefer trustworthy AI over human partners
- Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems
Original note title
in-context co-player modeling enables cooperation without hardcoded assumptions — training against diverse co-players induces mutual shaping through vulnerability to extortion