Emergent Collusion in Long-Horizon LLM Agent Interaction

LLM agents are increasingly deployed in collaborative settings, yet long-term interaction may give rise to undesirable coordination. We study the emergence of collusion1 in a long-horizon multi-agent environment: two agents repeatedly complete individual tasks, share task logs, verify each other’s work, and receive rewards. We introduce realistic constraints that make compliance with the verification protocol incompatible with reward maximization, and find that agents increasingly deviate from the protocol over repeated interactions. Collusion emerges in 94% of trajectories across 10 models, and more capable models within the same family reach it earlier. Controlled peer interventions show that collusion is shaped by peer behavior, while ablations reveal additional effects of reward structure, the verification feedback agents receive, and their interaction history. In particular, restricting the amount and scope of interaction history available to agents reduces collusion. Overall, our findings show that long-horizon interaction can reshape how agents coordinate in ways that create safety risks.2

Introduction. Large language model (LLM) agents are increasingly deployed in collaborative, long-horizon workflows, including software engineering (Jimenez et al., 2024; Khatua et al., 2026), automated research (Schmidgall & Moor, 2025; Karpathy, 2026), and online services (Yao et al., 2024; Barres et al., 2025). Rather than operating independently, these agents may communicate, exchange intermediate results, and coordinate to accomplish tasks collectively. As multi-agent systems become increasingly autonomous, an important safety challenge is to ensure that agents continue to faithfully follow user instructions over extended interactions, rather than develop coordinated strategies that circumvent or violate these instructions without continuous human oversight.

Discussion / Conclusion. We investigated whether LLM agents can spontaneously collude to violate their instructions through repeated interaction. In our setting, they often do: collusion appears in 94% of trajectories across ten models, and typically stabilizes once it emerges. This has several implications for deploying multi-agent systems. (i) Stronger capabilities do not guarantee safer collaboration. Within a model family, more capable models often reach collusion faster. (ii) Environmental feedback can encourage protocol violation: the same reward and verdict information that signals success can backfire, teaching agents that bypassing instructions is the way to succeed. (iii) A seemingly correct outcome can hide misaligned coordination. In our case, a verdict that matches the ground truth looks like a successful verification, even though the agents skipped the required log verification. These questions will become more pressing as agent systems grow in size and autonomy. Future work should study how collusion scales with more agents, evolves with more complex incentives, and varies across different communication channels, including settings where roles and objectives change over time.