Does automated evolution match human-built agent performance?
Can an agent improved through automated loops in 8 days generalize as well as an agent refined through human-driven R&D? This tests whether autonomous design iteration reaches human-level quality on tasks outside the training set.
The discussion's comparison: "On four held-out benchmarks spanning in- and out-of-distribution tasks, AIDE85 equals or surpasses AIDEhuman (section 3.3), a strong baseline developed through human-driven R&D (appendix B)." The evolved agent is labeled "AIDE85" in the excerpt, which does not define the label; I read it as the agent after the seven accepted rewrites, and that identification is mine.
The baseline is the point of the sentence. Seven rewrites accepted by an automated loop in 8 days are set against an agent that people improved by hand, and the paper's claim is that the loop's product is at least as good on tasks the loop did not select on. Read with Do AIDE2's improvements transfer to unseen tasks?, it says the automated route matched the human route on the held-out set.
What "equals or surpasses" does and does not say. It leaves room for ties on some benchmarks and wins on others, and the excerpt does not say which are which or by how much. It is a comparison to AIDEhuman, not to a test-time-search baseline at matched budget, so it does not stand in for the control in How should we measure gains from automatic harness evolution?. And it is a comparison with a human-driven R&D baseline, not with human–AI collaboration, so it does not test Can human-AI research teams improve faster than autonomous AI systems?: neither speed nor safety is compared in the excerpt, and the effort behind AIDEhuman (appendix B) is not given.
Inquiring lines that read this note 29
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do evolved harness optimizations generalize beyond their source benchmarks?- Why do evolved harnesses often fail to generalize beyond their training tasks?
- Do task-level outcomes provide sufficient supervision for harness evolution?
- How does test-time search budget compare to evolution gains under matched conditions?
- Do evolutionary discovery systems like FunSearch count as bounded or open-ended improvement?
- Does co-evolution empirically outperform single-entity self-improvement in standard evaluations?
- Do evolutionary archives let agents improve themselves without formal proof?
- How does the generation-verification gap shape evolutionary program search?
- How do evolutionary archives improve on single self-modification trajectories?
- How do evolutionary archives enable open-ended self-improvement without formal proofs?
- Can humans remain meaningfully in the loop as AI autonomy scales?
- How do human-in-the-loop systems compare to autonomous agent evaluators?
- How does automated mechanism discovery compare to human-led mechanistic research?
- Do efficiency gains in AI-assisted development stem from better tools or autonomous improvement?
- How much can computational speed and automation substitute for human scientific judgment?
- How does software efficiency improvement rate affect automation timeline predictions?
- How much of AI speedup evidence actually reflects invention versus adaptation?
- Can AI loops become self-sustaining if research automation keeps improving?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do AIDE2's improvements transfer to unseen tasks?
Whether gains from optimizing code on specific AI R&D tasks generalize to held-out benchmarks, including domains outside the selection distribution. This tests whether the agent learned reusable strategies or merely memorized task-specific fixes.
the transfer result this comparison sits on
-
How should we measure gains from automatic harness evolution?
Harness evolution itself runs a search loop, so reported improvements might come from more search rather than better design. What's the right way to measure whether the harness itself actually improved?
the control this comparison does not replace
-
Can human-AI research teams improve faster than autonomous AI systems?
Explores whether keeping humans actively involved in AI research collaboration accelerates paradigm discovery compared to fully autonomous self-improvement, and what safety advantages this preserves.
the claim that a human-collaboration baseline, not a human-only one, would test
-
Do stronger models always evolve harnesses better?
We explore whether base model capability predicts both the ability to write useful harness updates and the ability to benefit from them. The answer reshapes how we should allocate capability in self-evolving agent systems.
a model-tier dependence the excerpt does not address for a frontier agent
-
Can harness modules improve separately from benchmark data?
Does evolving harness components independently on out-of-distribution data, using contrasted success and failure trajectories, help distinguish reusable improvements from task-specific overfitting? This matters because current methods conflate general gains with benchmark adaptation.
Evidence for: ModularRSI's benchmark-disjoint, contrastive-trajectory, modular self-improvement supplies the mechanism behind AIDE2's gains over AIDEhuman
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement
- Rethinking the Evaluation of Harness Evolution for Agents
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
- Group-Evolving Agents: Open-Ended Self-Improvement via Experience Sharing
- RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts
- A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems
- ASI-Evolve: AI Accelerates AI
Original note title
on four held-out benchmarks the agent AIDE2 evolved equals or surpasses AIDEhuman, a strong baseline developed through human-driven R&D