When an AI rewrites its own code and keeps only the best versions, what tests decide which ones survive?
Which specific AI R&D tasks does AIDE2 benchmark itself against during selection?
This explores which named benchmark tasks AIDE2 uses to score its own self-rewrites when deciding which ones to keep, and the corpus doesn't list them.
This explores which named benchmark tasks AIDE2 uses to score its own self-rewrites when deciding which ones to keep. The corpus doesn't say. The closest statement is that AIDE2 proposed changes to its own code and "benchmarked the modified versions on AI R&D tasks", with no task names attached Can an AI agent reliably improve itself through hidden evaluation?. Anything more specific would be a guess.
The corpus does describe how the selection works. The scoring is hidden: a rewrite is kept only if it scores best on evaluations the proposing agent can't see. Each accepted rewrite then becomes the agent that proposes the next one. That produced seven accepted changes in eight days Can an AI agent reliably improve itself through hidden evaluation?. So the selection tasks act as a private judge, which may be why they aren't itemized in the summary.
The corpus does name a different set of tasks, and they are easy to mistake for the selection set. AIDE2's gains were checked on four held-out benchmarks covering machine learning, algorithm engineering, and physics-based weather forecasting. Weather forecasting is the one described as outside the selection task distribution Do AIDE2's improvements transfer to unseen tasks?. Held-out means they were not used for selection. The note implies the selection tasks sit closer to ML and algorithm work than to weather modeling, but it doesn't say which tasks they are.
The corpus does show what the selection produced. The seven kept rewrites clustered around search plateaus, context budgets, and wins that couldn't be trusted What problems did AIDE2's rewrites actually solve?. That points to selection tasks that stress long, budget-limited agent runs, though the notes don't confirm it. If you want a self-improving agent whose benchmarks are named, the Darwin Gödel Machine keeps or discards its own variants using SWE-bench and Polyglot Can AI systems improve themselves through trial and error?. That is a different system. The exact AIDE2 task list would be in the original paper's evaluation section, which these notes don't reproduce.
Sources 4 notes
An autonomous research agent proposed changes to itself, benchmarked variants on AI R&D tasks, and kept rewrites scoring best on evaluations the proposing agent could not see. Each accepted rewrite became the agent for the next iteration.
The paper reports that AIDE2's improvements transfer to four held-out benchmarks spanning machine learning, algorithm engineering, and physics-based weather forecasting—the last being outside the selection task distribution. This demonstrates transferable gains beyond overfitting to the selection set.
The seven accepted code changes clustered around search plateaus, context budgets, and untrustworthy wins—problems practitioners face building effective agents. This convergence suggests the autonomous loop discovered genuinely useful improvements rather than optimizing for scoring artifacts.
DGM replaces formal proofs with empirical benchmarking and maintains an evolutionary archive of agent variants, achieving 2.5× improvement on SWE-bench and 2.2× on Polyglot by discovering capabilities like better code editing and context management.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
- Recursive self-improvement of AI research agents
- The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators
- Self-Improvements in Modern Agentic Systems: A Survey
- PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents
- Hyperagents
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement