Does revealing rivals' capabilities reduce AI racing disaster risk?

Armstrong, Bostrom and Shulman build a game-theoretic model of n teams racing to build the first AI, where each team has a capability level c and chooses a safety level s between 0 (no precautions) and 1 (total safety); the team with the highest score c−s wins, and the winner then succeeds with probability s or causes an "AI-disaster" with probability 1−s. Solving for Nash equilibria across three information conditions, they report a "curious pattern": "the no-information case is always safer than the other two cases," while "for low μ and low e, the public information scenario can be safer than the private one." The paper's own framing is explicit about the surprise: "increasing the information available to all the teams... increases the risk. This is a special case of an information hazard."

The mechanism is that information changes what counts as a profitable risk. When a team knows only its own capability (private information), it only has an incentive to cut safety when its own capability is low, so a disastrously unsafe winner requires every team to happen to be low-capability at once — a probability the paper shows falls off "as 1/μn." When every team can see every other team's capability (public information), the winning team only need worry about the single closest rival; the paper's equilibrium sets the leader's safety at stop = Δ/e, where Δ is the capability gap to the second-place team, so a narrow gap between just two teams is enough to trigger risk-taking, and that narrow-gap probability falls off only "as 1/μ" — a much weaker safety margin at scale. The model also finds, more intuitively, that risk falls as μ rises (capability matters more than safety-skimping), as enmity e between teams falls, and — in the no-information and public-information cases — as the number of competing teams falls.

This gives a formal, decision-theoretic account of the race dynamic that Does slowing AI development actually prevent system failures? treats qualitatively: that paper's "pace debate" concerns how fast capabilities advance, while this model isolates a separate, counterintuitive lever inside any given pace — how much teams know about each other — that can move disaster risk independently of speed. Where the pace-debate note's residual risk comes from tight coupling between components, this paper's residual risk comes from the structure of what information is common knowledge among competitors, a distinct mechanism the vault had not yet held a note on.

The model is a stylized abstraction, not a measurement of any real AI race: capabilities are drawn independently and uniformly from a single interval [0, μ], every team shares the same enmity e toward every other team, teams are assumed to play exact Nash equilibria, and utility is renormalized to a clean 0/1 scale for disaster versus success. The paper gives no real-world estimate of μ, e, n, or disaster probability for any actual AI developer or lab. The implication the model supports is only structural: if real competitive dynamics even loosely resemble this setup, then interventions that increase mutual visibility into rivals' capabilities — public benchmarking, capability leaks, leaderboards — are a candidate channel for raising race-driven safety risk, which cuts against an intuition that more transparency is straightforwardly safer; the paper itself concludes only that this "may be of use in planning and regulating" AI development, not that it predicts any specific race's outcome.