Rival AI labs are racing each other, yet they might still have good reason to team up on safety rules — why would that ever make sense?

This explores why competing AI labs might rationally choose to cooperate on safety, and what makes that cooperation easier or harder. The short answer is that racing creates a classic collective action problem. Each firm cuts safety corners because slowing down alone is costly, and the result is an industry that is less safe than any single firm wanted. Can AI companies cooperate on safety despite competitive pressure? lays out five conditions that can tip firms toward cooperating: trust between them, gains everyone shares, low exposure to being exploited, little advantage from defecting, and losses everyone would suffer together. OpenAI makes the same argument for agentic AI specifically. Failures are rare but severe, while the payoff from deploying early is immediate, so firms rush agents into use before they've been properly checked. That's a race nobody chose on purpose Can competitive pressure force unsafe agentic AI adoption?.

What actually pushes a firm toward unsafe choices is often not what you'd expect. In a two-player race experiment, people's personal attitude toward risk didn't predict reckless development. Their competitive position did: falling behind made unsafe play more likely, and being ahead made it less likely. Players also tended to copy whatever their opponent had just done What actually drives unsafe choices in AI races?. So the danger sits mostly with the laggards, and recklessness spreads from one player to the next. Coordination helps because it takes away the panic of falling behind.

Here is the surprising part: more transparency between rivals doesn't automatically make things safer. In Armstrong, Bostrom and Shulman's game-theory model, when teams can see each other's capabilities, the leading team knows exactly how much room it has and is more willing to skip safety measures. When capabilities are hidden, every team has to hedge Does revealing rivals' capabilities reduce AI racing disaster risk?. Good coordination therefore depends on what gets shared, not just on sharing more.

Cooperation also hits obstacles outside the labs. The Congressional Research Service found that firms can legally share cybersecurity threat data under existing exemptions. Agreeing to slow frontier development together, though, has no such protection and could look like an antitrust violation What antitrust risks do different AI safety collaborations face?. Governments can also override industry plans. Amodei's pacing proposal was rejected by both Washington and Beijing within days Can AI safety pacing work without government cooperation?. That helps explain why groups like the Future of Life Institute argue companies can't police this alone and need binding oversight Can companies alone manage the risks of AI systems?, and why OpenAI now says international standards matter as much as alignment research for pacing the frontier Can global standards pace frontier AI as much as alignment research?.

The takeaway: firms have strong reasons to want coordination, because it is their way out of a trap they're all caught in. But voluntary coordination is fragile. Antitrust law limits it, it can collapse when governments compete with each other, and it can even backfire if the wrong information gets shared. That's why much of the debate has shifted from "will labs cooperate?" to "who outside the labs can make cooperation stick?"