Do AI researchers view automating AI research as a severe risk?
This inquiry examines whether leading researchers across labs and academia consider the automation of AI research itself to be among the most urgent risks facing the field, and whether their concern levels diverge by institution type.
The paper's central claim is that leading researchers, in labs and universities alike, treat automating AI research as one of the most serious risks. Of 25 researchers interviewed in August and September 2025, including participants from Google DeepMind, OpenAI, Anthropic, Meta, UC Berkeley, Princeton and Stanford, 20 "identified automating AI research as one of the most severe and urgent AI risks." The interviewees agreed that recursive improvement is possible but disagreed on timelines and governance. The excerpt also reports an "epistemic divide" between frontier companies and academic institutions. Every participant at a frontier company described active engagement with recursive-improvement scenarios and regular internal discussion, which they said is encouraged. Academic participants, by contrast, "often expressed limited consideration of these possibilities."
The reasoning runs through a feedback loop. The introduction describes an intelligence explosion, a scenario I.J. Good introduced in 1966, in which AI systems design successors that can design AI even better, and it cites Yudkowsky's definition in terms of the return on cognitive investment. The milestone it treats as critical is reached when AI systems can do the work of today's AI researchers, because such a system "could improve the development of" its successor. The excerpt states that AI systems "have not yet been able to recursively improve," so the risk is framed as a prospect rather than an observed event. It also notes, without tying it to the argument, that a version of GPT-5 won a gold medal at the International Math Olympiad in August 2025.
The nearest survey note, How soon do AI researchers expect artificial general intelligence?, measures timelines and extinction credence across 2,778 published researchers. This excerpt is narrower, with 25 interviews on one risk, and its disagreement sits on timelines and governance rather than on whether the loop is possible. Its divergence after the autonomous-developer milestone is the open trajectory that What bottlenecks define the path from AGI to superintelligence? says should be tracked through bottlenecks rather than bet on. The feedback-loop premise the excerpt assumes is what Can recursive self-improvement speed up the research process itself? examines: its case, against diminishing returns, is that automation improves artifacts while research efficiency stays fixed, a question this excerpt never addresses. Last, the excerpt reports agreement on "the possibility of recursive improvement" without separating bounded from open-ended forms, the split that Are self-refinement and recursive self-improvement actually the same thing? draws. The excerpt therefore cannot show whether its consensus covers one phenomenon or several.
The excerpt does not give the interview protocol, how the 25 were selected, how "most severe" was coded, or what the governance disagreements were. It is also cut off mid-quote, so the full account of the lab-academia divide is not visible. The causes it offers, "organizational culture, incentives, and proximity to cutting-edge capabilities," are stated as reasons for the divide, not tested. It uses "ASARA" without defining the term. The implication, at the strength the evidence allows, is that the risk is treated as live inside frontier labs and only partly taken up outside them. The 20-of-25 figure counts interviewees who identified the risk; it is not an estimate of how widely researchers hold the view, and the excerpt gives no grounds for judging academic attention as mistaken.
Inquiring lines that read this note 15
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What human oversight must AI research systems have?- What makes research tasks verifiable enough for AI automation?
- What governance approaches do researchers propose for automating AI research?
- How should labs measure their own AI systems' impact on research workflows?
- Why do AI researchers consider automating research itself a severe risk?
- How do researchers justify withholding AI from accountability-heavy work?
- Have AI researcher timelines shifted based on recent capability evidence?
- How does automated R&D affect the efficiency of the research process itself?
- What timeline disagreements emerge among researchers about autonomous AI development?
- How does automating research tasks change the pace of AI progress?
- Can partial automation in software research alone trigger runaway AI progress?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
How soon do AI researchers expect artificial general intelligence?
A survey of 2,778 AI researchers reveals how expert timelines for human-level AI have shifted over the past year, and what factors drive disagreement among specialists on this critical timeline.
a larger survey of published researchers on timelines and extinction credence; this study is narrower and isolates automating AI research
-
What bottlenecks define the path from AGI to superintelligence?
Rather than predicting when superintelligence arrives, this explores four candidate pathways—scaling, paradigm shifts, recursive improvement, and multi-agent collectives—and asks which frictions prove decisive or negligible in each route.
its open trajectory after the autonomous-developer milestone matches the landscape view of bottlenecks over a single timeline
-
Can recursive self-improvement speed up the research process itself?
Current AI research agents improve the artifacts they produce—faster training, cheaper inference—but not the pace of discovery itself. Can automating an agent's own code creation close that gap?
the feedback-loop premise the excerpt assumes; that note argues automation speeds outputs, not the research process, which the excerpt never addresses
-
Are self-refinement and recursive self-improvement actually the same thing?
The survey explores whether current AI systems using "self-X" vocabulary describe one unified phenomenon or fundamentally different processes with distinct evidence, theory, and risk profiles.
the bounded-versus-open-ended split the excerpt never draws, though its researchers agree recursive improvement is possible
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- AI Researchers' Views on Automating AI R&D and Intelligence Explosions
- PostTrainBench: Can LLM Agents Automate LLM Post-Training?
- RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts
- What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity
- Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report
- Fully Autonomous AI Agents Should Not be Developed
- Atria Dawn: The Dawn of Agentic Superintelligence
- Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?
Original note title
twenty of 25 interviewed AI researchers name automating AI research among the most severe risks — labs and academia diverge on engaging it