Can frozen models improve by evolving their harnesses?
DarwinX reports 17-point gains from selecting harness variants while keeping model weights frozen. The question is whether this improvement comes from population-level selection, the non-regression contract, the archive mechanism, or some combination of the three.
DarwinX's central claim is that an agent's harness, meaning its "prompts, tools, skills, and control flow," can be improved by treating self-evolution as selection over a population of harness variants while the model stays frozen. The abstract reports that "one loop adds about 17 points on average" across four benchmarks "that progressively separate the evolution signal from the test." Terminal-Bench 2.1 rises from 75.5 to 83.2% on a matched base and to 84.7% on a stronger one, TerminalWorld's held-out split reaches 68.3% (28/41), and WebArena-Infinity real-task pass@1 rises from 43.5% to 93.0% audit-clean. The discussion names Terminal-Bench 2.1 and WebArena-Infinity, both with GPT-5.5 frozen, as the strongest matched-model evidence.
The paper composes the method from three parts. A preserve-and-extend contract promotes a child harness "only when it improves on some task without giving up what its parent solved." An archive keeps alternative lineages so "complementary specialists can be inherited and recombined." A shared signal interface turns failure-, teacher-, and self-derived evidence into harness edits. Fitness comes from each benchmark's own verifier, with "no gold solutions, no hand-picked winners." The stated diagnosis is that single-lineage search "is path-dependent and local wins often regress other tasks." The introduction adds that most recent harness-evolution work shares one inner loop of batch rollouts, reflection, a bounded edit and a held-out or regression gate. DarwinX's departure sits at the population level and in the admission rule. A footnote says the selection language is "meant literally, not as a metaphor."
Against the vault, this carries the archive-plus-empirical-validation recipe from Can AI systems improve themselves through trial and error? to harnesses on a frozen model. That note records no non-regression admission rule, so the contract is the visible addition. It also bears on How should we measure gains from automatic harness evolution?. The "matched base" here holds the model fixed, which is a different control from holding search budget fixed, and the excerpt reports no test-time-search baseline. The disjoint TerminalWorld task set does address that note's second confound, evolution and evaluation sharing one benchmark. The small transfer result, 84.2% on SWE-bench Verified from a Terminal-Bench 2.1 harness, "inside a narrow band" around the fix-skill reference, fits Do harness edits learn reusable strategies or memorize task fixes?, though the excerpt never inspects what the edits contain.
The excerpt does not establish which part matters. No ablation of the contract, the archive or the signal interface appears, so the claim that population selection beats single-lineage search rests on the paper's motivation rather than on a comparison shown here. It also does not say what the 17-point average is taken over, and it gives no run counts, variance, compute or search budget. "Audit-clean" is undefined, and transfer is measured in one direction only. The defensible reading is that a population-level search with a non-regression gate can lift a frozen-model agent on verifier-scored tasks, with in-domain gains far larger than the transferred one. It is not yet evidence that any single component is necessary.
Inquiring lines that read this note 10
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How does harness optimization generalize across different model architectures and domains?- Which foundation model tiers most benefit from harness updates?
- How do evolved harness edits generalize across different benchmark domains?
- What makes a harness low-friction for model strategy?
- Why do useful harness updates often disappear during model evolution?
- How much does executor choice change a harness's actual performance?
- Can mid-tier models benefit more from harness improvements than frontier models?
- Does harness scaling represent a fundamentally different path than model scaling?
- How much does harness design contribute to reported model capability scores?
- How much of DarwinX's gain comes from maintaining an archive versus single-lineage search?
- Can harness evolution gains be distinguished from test-time search improvements on matched budgets?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can AI systems improve themselves through trial and error?
Explores whether replacing formal proof requirements with empirical benchmark testing enables AI systems to successfully modify and improve their own code iteratively, and what mechanisms prevent compounding failures.
same archive-plus-validation recipe, applied here to harnesses on a frozen model with an added non-regression admission rule
-
How should we measure gains from automatic harness evolution?
Harness evolution itself runs a search loop, so reported improvements might come from more search rather than better design. What's the right way to measure whether the harness itself actually improved?
DarwinX matches the model but the excerpt shows no matched-budget search baseline, so attribution to design stays open
-
Do harness edits learn reusable strategies or memorize task fixes?
When meta-agents evolve harnesses iteratively, do the persisted edits encode transferable procedures that solve new problems, or do they mostly cache shortcuts for already-solvable tasks? This matters because it determines whether harness evolution genuinely expands capability.
the far smaller transferred gain is compatible with task-specific edits, though DarwinX's edits are not inspected in the excerpt
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectories
Original note title
selection over a population of harness variants under a preserve-and-extend contract improves a frozen-model agent without training new weights