In a race to build AI, does punishing risky shortcuts actually stop people from taking them?
Does higher private risk eliminate unsafe play in competitive development races?
This explores whether raising the personal stakes of cutting corners (making unsafe development riskier for the developer itself) is enough to stop competitors from playing unsafely in an AI race, or whether something else drives that behavior.
This explores whether raising the personal cost of risky development is enough to stop competitors from cutting corners in an AI race. The short answer from the corpus: not on its own. The other side of the race seems to matter more than how much risk each player is willing to take. In a two-player race experiment, people's choice to develop unsafely didn't track their measured risk preferences. It tracked two things about the race. One was whether they were falling behind. The other was what their opponent had just done What actually drives unsafe choices in AI races?. Being ahead calmed people down. Lagging made them reckless, whatever their personal temperament. So making unsafe play riskier for the individual targets something that wasn't steering behavior in the first place. Being behind, and copying a rival's recklessness, can outweigh personal caution.
The game-theory side of the corpus points the same way, with a twist about information. In Armstrong, Bostrom and Shulman's race model, teams that know exactly how capable their rivals are become more willing to skip safety. When capabilities are hidden, every team has to hedge against being beaten, and that uncertainty acts as a safety buffer Does revealing rivals' capabilities reduce AI racing disaster risk?. Put this next to the experiment and a pattern emerges. Unsafe play is mostly about how you see your standing in the race. Anything that makes 'I'm behind' or 'they just defected' easier to see tends to push behavior toward danger.
That's why the corpus mostly treats racing as a collective action problem, not a matter of individual risk. Companies acting rationally underinvest in safety because each one's caution mainly benefits its rivals. What shifts the balance toward cooperation is how the race is structured: trust, shared gains, shared losses, and low payoffs for getting ahead Can AI companies cooperate on safety despite competitive pressure?. The Future of Life Institute takes this to its policy conclusion. If firms can't police themselves under competitive pressure, binding outside limits have to change the incentives Can companies alone manage the risks of AI systems?. One real-world test is sobering. Amodei's proposal to coordinate on pacing was reportedly rejected by both US and Chinese leadership within days, because competition between states outranked safety concerns Can AI safety pacing work without government cooperation?.
Even fixing the race wouldn't make failure impossible. Slowing development lowers risk in tightly connected systems but doesn't remove it, so governance also needs plans for stepping in and responding to harm Does slowing AI development actually prevent system failures?. One gap: these notes don't report an experiment that directly turned up the private penalty and measured whether unsafe play went to zero. What the corpus does show is that personal risk attitudes weren't what drove unsafe play. The surprising takeaway is that a cautious developer can still race recklessly when it's losing. If you want safer racing, change what competitors can see about each other and how much getting ahead pays, not how much any one of them has to lose.
Sources 6 notes
In a two-player AI race experiment, unsafe development followed the opponent's prior move and falling behind—not individual risk tolerance or measured risk preferences. Being ahead suppressed unsafe play while lagging behind amplified it, regardless of personal risk disposition.
A game-theoretic model shows that when AI teams have public information about competitors' capabilities, the winning team is more likely to skip safety measures. Private information creates a stronger safety buffer by requiring all teams to be weak simultaneously.
AI companies rationally underinvest in safety due to race dynamics, but five factors—trust, shared gains, low exposure, low advantage, and shared losses—can shift the equilibrium toward cooperation.
The Future of Life Institute argues that escalating AI incidents demonstrate private companies cannot self-police effectively, and calls for government-mandated limits on recursive self-improvement practices until safety research is complete, backed by hardware verification technology.
Trump and Xi Jinping both rejected Amodei's plan to coordinate AI safety measures immediately after its announcement, suggesting geopolitical incentives trump technological safety concerns among state leaders.
Show all 6 sources
Research shows slower pace lowers risk in complex coupled systems but does not prevent failures from occurring. When failure remains possible, governance must address intervention and harm response.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Role of Cooperation in Responsible AI Development
- Falling Behind Drives Unsafe Development in an Idealised AI Race Experiment
- Racing to the precipice: a model of artificial intelligence development
- Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
- Agentic Misalignment: How LLMs Could Be Insider Threats
- How AI Can Degrade Human Performance in High-Stakes Settings
- We Must Pace the Frontier
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems