When AI agents get smarter after automatic tune-ups, how much is real design progress versus just extra tries?
How much of harness-evolution gain comes from matched test-time search budgets?
This explores how much of the improvement from automatically evolving an agent's harness (its prompts, tools and scaffolding) would have appeared anyway if the agent had simply been given the same search and feedback budget.
This explores how much of the improvement from automatically evolving an agent's harness (its prompts, tools and scaffolding) would have appeared anyway if the agent had simply been given the same search and feedback budget. The corpus has no single percentage for this. It does give a clear way to measure it, and evidence that the share is larger than headline numbers suggest.
The measurement is a matched-budget baseline. How should we measure gains from automatic harness evolution? argues that evolved harnesses should be compared against plain task-level test-time search, with equal feedback and equal inference budget. Only the gap beyond that baseline counts as credit for the harness design. Anything below it is just more computation. The reason to worry is that Do harness edits learn reusable strategies or memorize task fixes? found the evolved edits look sensible, but most of them record fixes an agent could rediscover in a single rollout. They cache what is already within reach and rarely turn a hard failure into a success. Matched search would find those same fixes, so this is probably where much of the apparent gain goes.
Matched search is also a strong opponent, and its strength varies. Search budget scales like reasoning tokens, with steady returns that taper off (Does search budget scale like reasoning tokens for answer quality?). Evolutionary search at inference time beats best-of-N and sequential revision (Can evolutionary search beat sampling and revision at inference time?). How much search can find also depends on how the model was trained. Policies that have collapsed onto one answer stall where diversity-trained ones keep finding solutions (Should training maximize diversity when models feed into search?, Does reinforcement learning squeeze exploration diversity in search agents?). So a harness that beats a weak search baseline may lose to a well-run one. This fits the split in How do internal and external test-time scaling compare?: search extracts performance from capability the model already has. A harness only earns credit if it adds something search can't.
Some results look like real design value, though none is tested against matched search in the material here. Do AIDE2's improvements transfer to unseen tasks? reports gains that carry over to four held-out benchmarks, including weather forecasting outside the selection distribution. Memorized task-specific fixes would not transfer like that. Can explicit behavior maps help weaker planners compete with stronger models? shows a behavior-to-code map letting weaker planners match stronger models at code localization while using fewer tokens. That is a structural change to what the model sees, which extra sampling would not reproduce. The large self-improvement multipliers in Can AI systems improve themselves through trial and error? (2.5× on SWE-bench) and Can an AI system improve its own search methods automatically? (5×) don't say how much is design and how much is compute. That question is still open.
One more factor is who gets the gain. Do stronger models always evolve harnesses better? finds that the ability to write useful harness edits is flat across model tiers, but the ability to benefit from them peaks in mid-tier models. Weak models fail to invoke the harness, and strong models struggle to follow it faithfully. So the gain that survives a matched-budget comparison probably depends on the model tier. Search budget can help any model, whereas harness design mostly helps mid-tier ones, so the measured share would differ from one model to the next.
Sources 12 notes
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
Analysis of evolved harness trajectories shows rational, well-motivated edits across prompt and tool layers, but most persist fixes an agent could rediscover in a single rollout. Gains remain limited because memorized shortcuts cache what's already within reach rather than converting hard failures into successes.
Agentic deep research shows monotonic-to-diminishing-returns curves for search iterations, matching reasoning token scaling. This creates a new inference-compute axis: models can trade off reasoning budget against search budget to optimize answer quality.
Mind Evolution, an evolutionary search strategy using LLM-generated crossover and mutation with island model diversity, solves 98%+ of planning tasks and significantly outperforms best-of-N and sequential revision strategies while working directly in natural language without task formalization.
Vector Policy Optimization trains models to emit varied competent solutions rather than converging to one answer. This unlocks search procedures like evolutionary algorithms to explore and combine modes, solving problems that entropy-collapsed policies cannot reach at all.
Show all 12 sources
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Research shows test-time scaling methods split into internal (training models for autonomous reasoning) and external (inference-time search and verification). They complement rather than compete; internal builds capability while external extracts performance from existing capability.
The paper reports that AIDE2's improvements transfer to four held-out benchmarks spanning machine learning, algorithm engineering, and physics-based weather forecasting—the last being outside the selection task distribution. This demonstrates transferable gains beyond overfitting to the selection set.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
DGM replaces formal proofs with empirical benchmarking and maintains an evolutionary archive of agent variants, achieving 2.5× improvement on SWE-bench and 2.2× on Polyglot by discovering capabilities like better code editing and context management.
An outer loop successfully read inner loop code, identified bottlenecks, and generated new Python mechanisms at runtime, discovering combinatorial optimization and bandit methods that broke the inner loop's deterministic patterns and improved performance on GPT pretraining by 5x.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
- Does Thinking More always Help? Understanding Test-Time Scaling in Reasoning Models
- Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents