When an AI agent improves after you tweak its setup, is that smart design, or just extra computing power?
Can harness evolution gains be distinguished from test-time search improvements on matched budgets?
This explores whether the gains from evolving an agent's harness (its prompts, tools, and control flow) can be separated from the gains you'd get by spending the same compute searching harder at inference time.
This explores whether the gains from evolving an agent's harness (its prompts, tools, and control flow) can be separated from the gains you'd get by spending the same compute searching harder at inference time. The corpus says yes in principle, and it treats this as the standard to meet. Automatic harness evolution has to be compared against task-level test-time search under equal feedback and inference budgets, and only the margin beyond what matched search achieves belongs to the harness design itself (How should we measure gains from automatic harness evolution?). Without that comparison, you may just be measuring extra computation.
The baseline is a serious opponent because harness evolution is itself a search. DarwinX selects among a population of harness variants and lifts a frozen model by about 17 points on average (Can frozen models improve by evolving their harnesses?). The Darwin Gödel Machine keeps an archive of agent variants and reaches 2.5× on SWE-bench (Can AI systems improve themselves through trial and error?). Both try many candidates and keep the winners. That is the same shape as Mind Evolution, which does crossover and mutation at inference time and beats best-of-N and sequential revision on planning tasks (Can evolutionary search beat sampling and revision at inference time?). A large headline number can't tell you whether the harness holds a lasting insight or the loop simply ran many trials.
One fingerprint of search in disguise is what the edits contain. Analysis of evolved harnesses found that most edits persist fixes an agent could have rediscovered in a single rollout. They cache what's already within reach instead of turning hard failures into successes (Do harness edits learn reusable strategies or memorize task fixes?). Matched search would find those fixes anyway. The gains worth attributing to design are the ones that transfer, because per-task search can't carry over to a new task. ModularRSI evolves modules on benchmark-disjoint data and pools evidence across tasks before updating (Can harness modules improve separately from benchmark data?). AIDE2's gains hold on four held-out benchmarks, including weather forecasting outside its selection distribution (Do AIDE2's improvements transfer to unseen tasks?).
A second fingerprint is the kind of gain. Search buys accuracy by spending more compute. Several harness results spend less or shift the work elsewhere. Auto-research loops across many environments found four mechanisms that cut token traffic by 44.7–49.0% at comparable performance (Can agent harnesses be automatically optimized across many environments?). A behavior-to-code map let weaker planners match stronger models' code localization with 8–13% fewer planner tokens (Can explicit behavior maps help weaker planners compete with stronger models?). A stronger model built harnesses that nearly doubled a weaker model's Theory-of-Mind scores by moving unstable reasoning into deterministic code, not by encouraging longer reasoning (Can a stronger model lift a weaker one at test time without retraining?). Gains like these are hard to explain as more search.
The separation still won't be clean. Training and search gains can add together instead of substituting: Frontis-MA1 lifted Medal Average from 39.39% to 71.21% by combining both (Can training and search gains add together in program evolution?). So the goal is attributing a share to each, not choosing one. Model tier also skews the comparison. Skill at producing harness edits is flat across tiers, but the ability to benefit from them peaks in mid-tier models (Do stronger models always evolve harnesses better?). A matched-budget test at one tier could overstate or understate the harness's value. The corpus states the requirement clearly, but none of the other summaries report a head-to-head against a matched search baseline. That experiment is still the gap.
Sources 12 notes
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
DarwinX achieves average 17-point gains across benchmarks by evolving harness variants (prompts, tools, skills, control flow) under a preserve-and-extend contract while keeping the model frozen. Key evidence includes Terminal-Bench 2.1 rising to 84.7% and WebArena-Infinity reaching 93.0% audit-clean pass@1.
DGM replaces formal proofs with empirical benchmarking and maintains an evolutionary archive of agent variants, achieving 2.5× improvement on SWE-bench and 2.2× on Polyglot by discovering capabilities like better code editing and context management.
Mind Evolution, an evolutionary search strategy using LLM-generated crossover and mutation with island model diversity, solves 98%+ of planning tasks and significantly outperforms best-of-N and sequential revision strategies while working directly in natural language without task formalization.
Analysis of evolved harness trajectories shows rational, well-motivated edits across prompt and tool layers, but most persist fixes an agent could rediscover in a single rollout. Gains remain limited because memorized shortcuts cache what's already within reach rather than converting hard failures into successes.
Show all 12 sources
ModularRSI evolves harness modules independently using contrastive trajectories on benchmark-disjoint data, showing consistent gains across unseen tasks and domains. The approach isolates mechanism-level improvements from task-specific adaptation by aggregating evidence across tasks before updating components.
The paper reports that AIDE2's improvements transfer to four held-out benchmarks spanning machine learning, algorithm engineering, and physics-based weather forecasting—the last being outside the selection task distribution. This demonstrates transferable gains beyond overfitting to the selection set.
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
Frontis-MA1 trained a single model on four program-evolution operators (Draft, Improve, Debug, Crossover), then reused those operators in long-horizon evolutionary search. The result was complementary gains: learning and search improved performance together rather than substituting for each other, lifting Medal Average from 39.39% to 71.21% on MLE-Bench Lite.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable