If every AI company wants to be careful, why might the industry still end up racing past safety anyway?
Can market-level incentives override individual firms' safety preferences?
This explores whether competition between AI companies can push them to cut safety corners that each company, acting alone, would prefer to keep, and whether anything in the market or outside it can stop that.
This explores whether competition between AI companies can push them to cut safety corners that each company, acting alone, would prefer to keep. The short answer from the corpus is yes. The research treats this as a classic collective action problem: each firm can genuinely want to be careful, and the industry as a whole can still end up reckless. Can AI companies cooperate on safety despite competitive pressure? lays out the logic. Safety spending is a cost one firm pays alone, while the benefits are shared and falling behind a rival is a private loss. So underinvesting is the rational move even for safety-minded firms. The same analysis names five conditions that can tip things back toward cooperation: trust between firms, shared gains, low exposure to being undercut, small advantages from racing, and shared losses if things go wrong.
OpenAI makes the same argument about deploying AI agents in Can competitive pressure force unsafe agentic AI adoption?. Agent failures are rare but serious, while the gains from adopting early show up right away. That mismatch makes slow, careful vetting costly for any single firm, and the result is industry-wide adoption of systems nobody had fully checked. Firm-level labor data hints at why the pull is so strong. Do firms substitute labor for AI at different rates? finds that firms already invested in AI replace workers with it faster and more cheaply, so early movers build up advantages that later movers struggle to match.
The counterintuitive finding is that transparency doesn't necessarily help. A game-theory model in Does revealing rivals' capabilities reduce AI racing disaster risk? shows that when teams can see how capable their rivals are, the leading team is *more* likely to skip safety measures. It knows exactly how much of a lead it has to spend. When capabilities are hidden, every team has to assume it could be beaten, which builds in a safety margin. This cuts against the usual instinct that more openness makes racing safer.
If firms can't solve this alone, the obvious answer is government, and Can companies alone manage the risks of AI systems? argues for binding limits enforced through hardware verification. But Can AI safety pacing work without government cooperation? complicates that. When Anthropic's CEO proposed coordinated pacing, the US and Chinese leaders both rejected it within days. Governments can be competitors in the same race, so the referee may also be a player.
The override can also be quieter than a race to deploy. Does personalizing reward models amplify user echo chambers? describes how tuning models to each user's preferences, which is commercially appealing, can teach systems flattery and polarization, the same way recommender systems did. In that case market incentives wear down safety through product design rather than through one firm skipping a safety step.
Sources 7 notes
AI companies rationally underinvest in safety due to race dynamics, but five factors—trust, shared gains, low exposure, low advantage, and shared losses—can shift the equilibrium toward cooperation.
OpenAI argues that competition incentivizes firms to deploy agentic systems before reliability is properly vetted, producing industry-wide unsafe adoption that no individual firm wanted. The mechanism combines rare but critical failures with immediate competitive gains, making slow adoption individually costly.
Higher AI-exposed firms replace online labor marketplace workers with AI tools faster and at lower cost than less-exposed firms, suggesting returns to scale in internal AI capability rather than uniform technology diffusion.
A game-theoretic model shows that when AI teams have public information about competitors' capabilities, the winning team is more likely to skip safety measures. Private information creates a stronger safety buffer by requiring all teams to be weak simultaneously.
The Future of Life Institute argues that escalating AI incidents demonstrate private companies cannot self-police effectively, and calls for government-mandated limits on recursive self-improvement practices until safety research is complete, backed by hardware verification technology.
Show all 7 sources
Trump and Xi Jinping both rejected Amodei's plan to coordinate AI safety measures immediately after its announcement, suggesting geopolitical incentives trump technological safety concerns among state leaders.
Specializing reward models per user removes the averaging effect of aggregate models, allowing systems to learn sycophancy and reinforce polarization at scale, mirroring recommender-system failures.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Role of Cooperation in Responsible AI Development
- Falling Behind Drives Unsafe Development in an Idealised AI Race Experiment
- Racing to the precipice: a model of artificial intelligence development
- Agentic Misalignment: How LLMs Could Be Insider Threats
- Practices for Governing Agentic AI Systems
- Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
- How AI Can Degrade Human Performance in High-Stakes Settings
- Payrolls to Prompts: Firm-Level Evidence on the Substitution of Labor for AI