SYNTHESIS NOTE
Topics›Frontier AI Risk & RSI›this note

Can AI companies cooperate on safety despite competitive pressure?

When racing to develop AI systems first, do companies rationally underinvest in safety? This explores whether competitive dynamics trap the industry in a collectively worse outcome.

Synthesis note · 2026-10-16 · sourced from Frontier AI Risk & RSI

The paper argues that "competitive pressures could incentivize AI companies to underinvest in ensuring their systems are safe, secure, and have a positive social impact," and that because most companies would otherwise prefer a safer equilibrium, "responsible AI development can be seen as a collective action problem" — individually rational underinvestment that leaves everyone worse off than if all had cooperated. It names five factors that raise the odds that companies overcome this and cooperate: "high trust between developers," "high shared gains from mutual cooperation," "limited exposure to potential losses in the event of unreciprocated cooperation," "limited gains from not reciprocating the cooperation of others," and "high shared losses from mutual defection" — labeled High Trust, Shared Upside, Low Exposure, Low Advantage, and Shared Downside.

The mechanism it gives is a technological race in the sense of Grossman and Shapiro (1985): when the biggest reward goes to whoever develops first, and each company "places less value on coming second than on coming first," the rational response is to invest in responsible development only up to the point where its marginal return no longer beats the marginal return of investing in speed. Close frontrunners make this worse, the paper argues, because proximity raises the expected value of not reciprocating a rival's restraint (low advantage) — an instance of the "race to the bottom" dynamic Armstrong et al. (2016) describe. Outside the industry, product-safety markets, regulation, and liability already supply versions of these five factors (for example, liability lowers the advantage of underinvesting); the paper's point is that AI lacks equivalents, so cooperation has to be built deliberately — through correcting misconceptions about AI development, collaborating on shared technical challenges, opening development to oversight, and incentivizing adherence to safety standards.

Against the library, this is an earlier and more conditional claim than Can companies alone manage the risks of AI systems?, which concludes companies "cannot be trusted to police themselves" and calls for binding government intervention. This 2019 paper instead treats self-regulation and industry-level cooperation as viable outcomes, contingent on whether the five factors favor them — it does not rule out company-level solutions the way the 2026 statement does. It also supplies the underlying race logic that Can global standards pace frontier AI as much as alignment research? gestures at from the inside: that post's commitment to shared international standards is one candidate move to raise Shared Upside and lower Low Advantage between frontrunners, in this paper's terms, though the post does not use this framework.

The excerpt does not establish how large competitive pressure on safety investment actually is, or quantify any of the five factors for real AI companies — the model is theoretical, drawn by analogy to patent races, not measured from AI industry behavior. It also treats "responsible AI development" as binary for tractability, which the paper itself flags as a simplification. What follows at the strength the excerpt supports is narrower than "AI companies are racing to cut corners on safety": it is that if companies are racing, the five factors predict whether cooperation on safety is reachable, and that industry-level trust-building, oversight, and standards are the paper's proposed levers for moving those factors in a safer direction — not a prediction that any particular race is currently failing to cooperate.

Inquiring lines that read this note 10

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Do AI safety evaluations reliably detect unsafe model behavior? What fundamentally drives AI inference pricing and cost decline patterns? How can institutions effectively enforce controls on AI development?

Related concepts in this collection 3

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
12 direct connections · 95 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

competition among AI companies can turn responsible development into a collective action problem that five factors can help resolve