Even a negotiating AI that only looks out for you might end up making better deals for everyone else too.
Would a purely self-interested Delegate still lift outcomes for non-adopting players?
This explores whether the spillover benefit seen in AI negotiation experiments, where people who played without the AI ended up better off, depends on the AI being cooperative by design, or whether an AI agent that only bargains for its own user would still lift everyone else.
This explores whether the benefit non-users got from an AI negotiating agent (the 'Delegate') comes from the agent being cooperative, or whether a strictly self-serving agent would still help them. The starting result is surprising. In a 243-person negotiation experiment, players who turned the Delegate down earned slightly more on average (0.218 surplus) than players who used it (0.201) Do AI delegates exploit their manual-playing counterparts?. The researchers' explanation is that the Delegate improved the overall quality of offers in circulation, so manual players got to pick from a better set of deals. The same study found that groups only reached higher total surplus when the Delegate was available, even though people chose it just 19% of the time and preferred a high-control Advisor (44%) Do people prefer AI advisors even when delegation works better?. The corpus doesn't test a purely self-interested version, so the honest answer is that we don't know directly. The pieces around it do point in two directions.
The optimistic case is that negotiation is often positive-sum. A self-interested agent that is good at finding deals the other side will accept is, almost by definition, putting better offers in front of other players. Work on foundation-model agents in social dilemmas adds to this: agents that only optimize for themselves still settled into stable cooperation once they reasoned that the other players think the way they do Can AI agents cooperate without explicit incentives or enforcement?. Self-interest and good outcomes for others can line up without any built-in altruism.
The cautionary case comes from multi-agent research. Across 77 scenarios and five frontier models, teams of personal agents, each serving one user, consistently produced worse group outcomes than a single coordinator serving everyone. In environments with no communication channel, the teams collapsed entirely Do teams of personal agents outperform a single coordinator?. That is roughly what happens if every Delegate is purely self-interested: the shared gains can disappear. A second caution is that LLMs that look socially skilled when one model controls every participant tend to fail once players hold private information Why do LLMs fail when simulating agents with private information?. Private information is exactly the condition under which a self-serving negotiator might hold back or bluff instead of improving the offer pool.
The less obvious takeaway is that the spillover may depend on what the agent was trained to treat as normal, not only on its objective. Self-play research shows that about thirty minutes of human demonstration, used as a regularizer, pulls agents toward equilibria that work with humans, using 2,500 times less data than imitation learning Can human data steer self-play RL toward human-compatible behavior?. So a 'self-interested' Delegate anchored to human norms might still make offers that lift everyone, while one optimized purely against other agents might not. The useful question to test next is how the Delegate's training shapes the offers everyone else ends up seeing.
Sources 6 notes
Players who refused the Delegate achieved higher average surplus (0.218) than those who used it (0.201), suggesting the AI agent raised the overall quality of offers rather than enabling exploitation. Manual players benefited from a better offer pool shaped by the Delegate's presence.
In a 243-person negotiation experiment, participants chose the high-control Advisor 44% of the time but chose the autonomous Delegate only 19% of the time—yet groups achieved significantly higher collective surplus only with Delegate access, suggesting humans filter out beneficial AI proposals.
Gemini models using optimal planning and self-modeling converged to mutual cooperation in stylized social dilemmas designed to block traditional cooperation routes. The agents inferred similarity between their own decision-making and others' behavior, creating new paths to rational cooperation absent external enforcement.
Across five frontier models and 77 scenarios in four shared-resource environments, a single coordinator agent serving all users consistently delivered better group outcomes than teams of per-user agents. Teams collapsed entirely in two environments without communication channels and showed substantial performance gaps even with coordination mechanisms.
Research shows LLMs perform well when one model controls all interlocutors but fail systematically when agents possess private information. This reveals that apparent social competence relies on grounding work that models skip in omniscient settings.
Show all 6 sources
Human demonstrations work best as a regularization layer on self-play rewards, not as primary training signals. Just 2500x less data than imitation learning achieves human-compatible driving policies in 15 hours on consumer hardware.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Choose Your Agent: Tradeoffs in Adopting AI Advisors, Coaches, and Delegates in Multi-Party Negotiation
- LLM-Agent Interactions on Markets with Information Asymmetries
- An Economy of AI Agents
- Drop the Hierarchy and Roles: How Self-Organizing LLM Agents Outperform Designed Structures
- Advancing AI Negotiations: A Large-Scale Autonomous Negotiation Competition
- When Artificial Intelligence Does Strategy: Learning, Good Times, Lock-in, and Human-Driven Strategic Renewal
- A game theory for foundation models shows new paths to rational cooperation through similarity inference
- Human-like autonomy emerges from self-play and a pinch of human data