Does Sakana's AI Scientist deliver autonomous research without human help?

Beel and Kan et al. test Sakana's claim that the AI Scientist "can autonomously run the entire life cycle of machine learning research without any human intervention except for initial preparation." On one template (Green Recommender Systems, FunkSVD trained on MovieLens-100k) they conclude the system "does not yet fulfill its promises." Its literature review relies on "simplistic keyword searches rather than profound synthesis", so several generated ideas were wrongly judged novel, including micro-batching for stochastic gradient descent. Five of twelve proposed experiments (42%) "failed due to coding errors", and those that ran often gave "logically flawed or misleading results", one reporting accuracy gains while using more computation against an energy goal. Manuscripts had a median of five citations, only five of 34 from 2020 or later, and some contained hallucinated numerical results, missing figures and placeholder text such as "Conclusions Here."

The authors locate part of the limit in the setup. They "initially assumed the AI Scientist could autonomously conduct research based solely on a prompt," but it "requires a user-defined 'template'": a pipeline in a special format, plus seed ideas whose interestingness, feasibility and novelty scores "had no apparent impact on the AI Scientist's processing." Code changes are small, with each iteration adding "only 8% more characters on average," which they read as "limited adaptability." Its reviews "focus on surface-level critiques, while failing to detect deeper methodological flaws." These verdicts rest on the authors' own reading of the generated outputs; the excerpt describes no independent benchmark or formal check.

This qualifies the closed-loop optimism in Can automated review loops handle AI-generated research at scale?, which reports that aiXiv's review-refine loop improves proposal and paper quality through iteration. The AI Scientist is a different system, so this is not a refutation, but it sharpens the question for automated venues: whether review is deep enough to catch methodological flaws, not only whether it runs. The failures are also the checkable kind that Can separating judgment from verification improve research paper reliability? moves out of model judgment, though the excerpt says nothing about how the AI Scientist's own checks work. The template requirement is where How much guidance do AI systems need to conduct research independently? would place the autonomy limit. The excerpt's point that second-level reviewers must "look beyond surface-level analyses" also fits How should AI agents and humans divide research tasks?, which finds humans keeping most final decisions, from a different evidence base.

The excerpt does not show how far these results travel. It covers one domain, one dataset, two seed ideas and one template, and the authors concede these "may affect generalizability and reproducibility." Twelve experiments is a small sample, and nothing here shows the system in other fields or whether later versions fix these faults. The authors hold two verdicts at once: the system "produces complete research manuscripts with minimal human intervention," and it fails the checks above. Their forecast that the shortfalls are "technical hurdles rather than fundamental barriers" is an expectation, not a measurement. The supportable reading is narrower: for this topic and template, output needs human checking of experiments, citations and novelty claims, and these results do not by themselves estimate a general failure rate.