How much guidance do AI systems need to conduct research independently?

ASI-Bench makes a design claim about what a research benchmark has to vary. The abstract argues that artificial superintelligence requires AI to move from mastering existing knowledge toward "exploring the unknown, creating new knowledge, and turning new ideas into verifiable results," while existing benchmarks "primarily test whether AI can produce correct answers based on learned knowledge, or whether it can complete tasks under extensive human guidance." Neither shows whether AI can conduct research when "both the problem and the path to a solution are open-ended." The authors present ASI-Bench as the first benchmark to jointly evaluate innovative exploration and autonomous scientific execution, and the first to "progressively withdraw human methodological guidance within the same research project."

The mechanism is the withdrawal itself. The benchmark has 60 project-level tasks across 11 scientific domains, and it "progressively reduces methodological guidance to test whether AI can independently select methods, conduct research, and produce verifiable results." The design implies that a change in performance between levels can be read as an effect of the removed guidance, since the project stays the same. The discussion points to a "B1–B4 structure" and domain-level analyses that enable "controlled comparisons across models and agents, while revealing where progress occurs and where human methodological guidance remains necessary." The question becomes a curve (how far capability extends as guidance recedes) instead of a single pass-or-fail score. The paper backs the tasks with over 40 experts, 31,000+ human hours, expert review, AI-assisted auditing, sandbox execution, and scorer validation.

This sits beside Do automated benchmarks hide what frontier AI systems can really do? as a different answer to the same complaint. That note says benchmarks favor precisely specified, auto-gradable tasks and turns to small-sample qualitative analysis of messy real work. ASI-Bench keeps the benchmark form, with sandboxes and validated scorers, and makes the level of specification the experimental variable. It also offers a second axis for testing Where does AI assistance become unreliable in research?. That note locates the boundary by lifecycle stage, and a guidance-withdrawal design would locate it by how much methodology the human supplies. Its pairing of exploration with execution roughly parallels What capabilities do AI systems need for autonomous science?, though the excerpt does not say how the two line up.

The excerpt reports no results. It gives no scores, no model comparisons, no definition of what B1 through B4 mean, no account of how guidance is operationalized at each level, and no description of how "innovative" exploration is judged. It cannot say whether capability degrades gradually or abruptly as guidance recedes; the introduction poses that as its question and leaves it open. The discussion also concedes that "no fixed benchmark" can represent the full breadth of meaningful, verifiable problems. What follows at this strength is that the guidance-withdrawal design is a well-motivated way to measure autonomy, and any claim about where human methodological guidance stays necessary, which bears on Can human-AI research teams improve faster than autonomous AI systems?, has to wait for the results.