Statement: We must pressure AI companies to immediately limit the use of recursive self improvement
Source: Future of Life Institute · 2026-09-28
“Today, for the first time, heads of state recognise that the risks of AI cannot be managed by the companies alone. Every day, new incidents of increasing severity show that there is an urgent and vital need for independent, binding oversight of AI.
“Last week, Anthropic CEO Amodei and OpenAI CEO OpenAI proposed on-site inspectors to test the safety of their systems, and world leaders have called for the creation of an international governance institution. These are excellent first steps, but we need more – and they absolutely must not include governments building a liability shield for these companies.
“As UN Human Rights Chief Türk has urged, countries should put pressure on companies to immediately limit the use of recursive self improvement until the research to make it safe has been done. In parallel, countries should ramp up investment in hardware verification technology that can enable a future international AI deal involving both China and the U.S.
“It is now more clear than ever that private AI companies cannot be trusted to police themselves. Government intervention is urgently required to halt the ongoing race-to-replace, and help build a prohuman future with AI for everyone.”
Lines of inquiry this paper opens 24
Research framings built by reading the notes related to this paper — the questions it feeds into.
How can humans maintain effective oversight as AI systems scale?- What mechanisms could concentrate AI power in too few hands?
- Can third-party evaluators monitor AI systems without regulatory teeth?
- How does speed of AI development threaten human ability to intervene?
- Are AI companies already implementing slowdowns in development as claimed?
- Can regulators adapt fast enough if they wait for risk evidence to emerge?
- What biological and autonomy risks does Amodei expect to follow cyber risks?
- What happens to self-regulation when a company's IPO plans conflict with safety?
- Who should design and enforce measures that slow AI capability development?
- How does staged access to powerful AI reduce dual-use harm?
- What concrete baseline safeguards should global frontier AI standards actually require?
- Who has authority to halt a deployed AI system causing harm?
- How would hardware verification technology enable international agreements on AI safety?
- Would fragmented national AI standards make comparing safety evidence harder across labs?
- Can standards enforcement prevent any single nation from accelerating unsafe AI research?
- Can error visibility alone improve AI system safety without containment?
- Does slowing AI development reduce risk or just delay it?