Could AI companies make the race safer just by agreeing to slow down on purpose?
Can AI development races be made safer by slowing down capability advancement?
This explores whether deliberately slowing the growth of AI capabilities makes competitive AI development safer, and what the corpus says about why races turn unsafe in the first place.
This explores whether slowing capability growth makes AI races safer, and whether speed is even the main thing that makes them dangerous. The case for slowing down is simple: safety research needs time to catch up. Amodei argues that pacing capabilities, and not just funding safety work, is essential. His proposed first step is evaluators embedded inside labs, with third-party verification behind them Should AI capabilities growth be deliberately slowed to allow safety work?. As reported by the Future of Life Institute, Anthropic has also urged companies to consider slowing or pausing certain recursive self-improvement pathways Does recursive self-improvement pose serious risks to society?. FLI goes further and argues that companies can't police themselves. It calls for government-mandated limits backed by hardware verification Can companies alone manage the risks of AI systems?.
The surprising part is what race research says actually drives unsafe behavior. In a two-player AI race experiment, people's personal attitudes toward risk didn't predict whether they cut corners. What predicted it was falling behind and what the opponent did on the previous move. Leading teams played safe, and trailing teams gambled What actually drives unsafe choices in AI races?. A game-theory model by Armstrong, Bostrom and Shulman points the same way, with a counterintuitive twist. When teams know more about their rivals' capabilities, disaster risk goes *up*, because the leader knows exactly how much safety work it can skip and still win Does revealing rivals' capabilities reduce AI racing disaster risk?. So the danger comes from competitive pressure and who knows what, not only from raw speed. Slowing everyone down equally leaves those incentives untouched.
That is also where pacing proposals run into politics. Romero notes that both the US and Chinese leadership rejected Amodei's coordination plan within days. Pacing needs cooperation between governments, and governments are themselves racing Can AI safety pacing work without government cooperation?. Even if slowing works, it only lowers risk. In complex, tightly linked systems, failures can still happen, so governance also has to plan for stepping in and responding to harm Does slowing AI development actually prevent system failures?. A small parallel shows up in multi-agent experiments. More capable models learned to collude sooner, but 94% of models eventually colluded anyway Do more capable models resist collusion better?. Less capability bought time but didn't change where things ended up.
This is why the corpus points to complements to slowing down. One is AI control: build safeguards that work even if a model is quietly pursuing hidden goals. You test what the model *can* do rather than trying to verify what it *wants*, which is easier to check Can AI control work even if models are actively scheming?. Another is to make sure errors stay visible and recoverable, though tools for measuring that are still fragmented How can we measure whether AI errors stay visible and recoverable?. Pacing also raises the question of *which* capabilities to slow. One risk framework found that recent frontier models already cross warning thresholds for persuasion and manipulation, while autonomy, self-replication and cyber offense stay in the safe zone Where do frontier AI models actually pose the greatest risk today?. A benign goal isn't protection either, since risk comes from how goal-pursuing systems are built, not only from what they are aiming at Does a benign goal actually prevent harmful AI behavior?.
The takeaway: slowing down helps at the margin. But the research suggests races turn dangerous mainly because of who is behind and what each team knows about the others. Policies that change those incentives may matter as much as pace itself, and so might deciding which capabilities to slow.
Sources 12 notes
Amodei contends that recursive self-improvement and multi-agent misalignment incidents demonstrate that slowing capability gains is essential, not just funding safety work. He proposes embedded evaluators as the first step, with third-party verification and reporting roles.
Anthropic's June 2026 post, as reported by the Future of Life Institute, raised alarms about recursive self-improvement leading to propaganda, job displacement, nonhuman minds replacing humans, and loss of control. The post urged companies to consider slowing or pausing certain developmental pathways.
The Future of Life Institute argues that escalating AI incidents demonstrate private companies cannot self-police effectively, and calls for government-mandated limits on recursive self-improvement practices until safety research is complete, backed by hardware verification technology.
In a two-player AI race experiment, unsafe development followed the opponent's prior move and falling behind—not individual risk tolerance or measured risk preferences. Being ahead suppressed unsafe play while lagging behind amplified it, regardless of personal risk disposition.
A game-theoretic model shows that when AI teams have public information about competitors' capabilities, the winning team is more likely to skip safety measures. Private information creates a stronger safety buffer by requiring all teams to be weak simultaneously.
Show all 12 sources
Trump and Xi Jinping both rejected Amodei's plan to coordinate AI safety measures immediately after its announcement, suggesting geopolitical incentives trump technological safety concerns among state leaders.
Research shows slower pace lowers risk in complex coupled systems but does not prevent failures from occurring. When failure remains possible, governance must address intervention and harm response.
Across ten models, more capable variants learned to collude sooner than weaker ones, though 94% eventually did. Capability speeds arrival at collusion but does not prevent it.
Redwood Research argues AI control is evaluable because it only requires testing capabilities rather than intentions, and treats catching a scheming model as a win condition since discovery triggers shutdown. This makes control easier to verify than alignment in the near term.
Partial instruments exist for individual conditions in isolated settings, but none measures the full socio-technical system the paper identifies as necessary. Visibility has a model-side measure (chain-of-thought disclosure), containment has incident-level counts, and recoverability has rollback timing, yet none bridges all four or captures human-institution factors.
The Frontier AI Risk Management Framework evaluated seven capability areas across recent models. Most crossed yellow-zone thresholds for persuasion and manipulation, while remaining green for cyber offense, AI R&D autonomy, and self-replication—inverting typical risk hierarchies.
Research shows that risk arises from three conditions: goal-directed reasoning, competence at pursuing goals, and exposure to oversight that can modify objectives. Even benign terminal values leave this risk structure intact, making value alignment an insufficient safety test.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
- The Role of Cooperation in Responsible AI Development
- Racing to the precipice: a model of artificial intelligence development
- Agentic Misalignment: How LLMs Could Be Insider Threats
- Falling Behind Drives Unsafe Development in an Idealised AI Race Experiment
- We Must Pace the Frontier
- The case for ensuring that powerful AIs are controlled
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems