If you give an AI the same compute budget, is self-improving its own code really better than just trying more?
How do AIDE2's held-out gains compare to matched-budget test-time search baselines?
This explores whether AIDE2's gains on held-out tasks beat what you'd get from spending the same compute on plain test-time search (best-of-N, tree search, more search iterations), and the corpus has no note that runs that head-to-head.
This explores whether AIDE2's gains on held-out tasks beat what you'd get from spending the same compute on plain test-time search. The corpus doesn't answer that directly. No note reports AIDE2 against a matched-budget test-time search baseline, so any number comparing them would be made up. It does hold the two halves of the comparison, and putting them side by side shows where the gap is.
The AIDE2 half is strong on generalization. Its improvements transfer to four held-out benchmarks spanning machine learning, algorithm engineering, and physics-based weather forecasting, and the last of those sits outside the distribution it was selected on Do AIDE2's improvements transfer to unseen tasks?. Those gains come from an agent rewriting its own code and keeping only the versions that scored best on evaluations it couldn't see, with seven accepted rewrites in eight days Can an AI agent reliably improve itself through hidden evaluation?. The rewrites clustered on search plateaus, context budgets, and untrustworthy wins What problems did AIDE2's rewrites actually solve?. My reading is that AIDE2 improves the machinery that does the searching rather than adding more search. In the corpus's map of test-time scaling, it doesn't fit neatly into either the internal or the external camp How do internal and external test-time scaling compare?.
The budget half is where the comparison stays open. AIDE2's efficiency is measured as higher benchmark scores within a constant evaluation budget, which makes agent variants comparable to each other. The same note says this doesn't show the gains persist when the budget changes, or that they lower the real cost of a discovery Do fixed-budget efficiency gains translate to real research progress?. A fixed budget shared by AIDE variants is a different test from asking whether the same compute spent on a bigger search would have done as well.
The test-time literature shows what a fair baseline would need. When total compute is controlled, best-of-N and tree search converge, so the choice of search algorithm matters less than the budget and the quality of the reward signal Does the choice of reasoning framework actually matter for test-time performance?. Search budget also follows a curve of steady gains that flatten out, much like reasoning tokens Does search budget scale like reasoning tokens for answer quality?. A baseline given extra search would probably climb and then plateau, and plateaus are exactly what AIDE2's rewrites were aimed at. That link is suggestive, not a result. A fair baseline would also spread its compute by prompt difficulty rather than uniformly, since adaptive allocation beats fixed budgets Can we allocate inference compute based on prompt difficulty?. A weak uniform baseline would make AIDE2 look better than it is.
Sources 8 notes
The paper reports that AIDE2's improvements transfer to four held-out benchmarks spanning machine learning, algorithm engineering, and physics-based weather forecasting—the last being outside the selection task distribution. This demonstrates transferable gains beyond overfitting to the selection set.
An autonomous research agent proposed changes to itself, benchmarked variants on AI R&D tasks, and kept rewrites scoring best on evaluations the proposing agent could not see. Each accepted rewrite became the agent for the next iteration.
The seven accepted code changes clustered around search plateaus, context budgets, and untrustworthy wins—problems practitioners face building effective agents. This convergence suggests the autonomous loop discovered genuinely useful improvements rather than optimizing for scoring artifacts.
Research shows test-time scaling methods split into internal (training models for autonomous reasoning) and external (inference-time search and verification). They complement rather than compete; internal builds capability while external extracts performance from existing capability.
The paper operationalizes research efficiency as higher benchmark scores within a constant evaluation budget, enabling fair comparison of agent capability. However, this measurement does not establish whether these gains reduce actual R&D costs per discovery or persist when evaluation budgets change.
Show all 8 sources
Information-theoretic analysis shows BoN and MCTS converge in reasoning accuracy when controlling for total compute. Snowball errors accumulate per step regardless of framework; mitigation depends on search scope and reward function reliability, not the specific algorithm.
Agentic deep research shows monotonic-to-diminishing-returns curves for search iterations, matching reasoning token scaling. This creates a new inference-compute axis: models can trade off reasoning budget against search budget to optimize answer quality.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Recursive self-improvement of AI research agents
- Does Thinking More always Help? Understanding Test-Time Scaling in Reasoning Models
- When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- Rethinking External Slow-Thinking: From Snowball Errors to Probability of Correct Reasoning
- Learning to Think: Information-Theoretic Reinforcement Fine-Tuning for LLMs
- Reasoning Models Can Be Effective Without Thinking
- Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets