SYNTHESIS NOTE
Topics›Agent Harness›this note

Can frozen models improve by evolving their harnesses?

DarwinX reports 17-point gains from selecting harness variants while keeping model weights frozen. The question is whether this improvement comes from population-level selection, the non-regression contract, the archive mechanism, or some combination of the three.

Synthesis note · 2026-09-25 · sourced from Agent Harness

DarwinX's central claim is that an agent's harness, meaning its "prompts, tools, skills, and control flow," can be improved by treating self-evolution as selection over a population of harness variants while the model stays frozen. The abstract reports that "one loop adds about 17 points on average" across four benchmarks "that progressively separate the evolution signal from the test." Terminal-Bench 2.1 rises from 75.5 to 83.2% on a matched base and to 84.7% on a stronger one, TerminalWorld's held-out split reaches 68.3% (28/41), and WebArena-Infinity real-task pass@1 rises from 43.5% to 93.0% audit-clean. The discussion names Terminal-Bench 2.1 and WebArena-Infinity, both with GPT-5.5 frozen, as the strongest matched-model evidence.

The paper composes the method from three parts. A preserve-and-extend contract promotes a child harness "only when it improves on some task without giving up what its parent solved." An archive keeps alternative lineages so "complementary specialists can be inherited and recombined." A shared signal interface turns failure-, teacher-, and self-derived evidence into harness edits. Fitness comes from each benchmark's own verifier, with "no gold solutions, no hand-picked winners." The stated diagnosis is that single-lineage search "is path-dependent and local wins often regress other tasks." The introduction adds that most recent harness-evolution work shares one inner loop of batch rollouts, reflection, a bounded edit and a held-out or regression gate. DarwinX's departure sits at the population level and in the admission rule. A footnote says the selection language is "meant literally, not as a metaphor."

Against the vault, this carries the archive-plus-empirical-validation recipe from Can AI systems improve themselves through trial and error? to harnesses on a frozen model. That note records no non-regression admission rule, so the contract is the visible addition. It also bears on How should we measure gains from automatic harness evolution?. The "matched base" here holds the model fixed, which is a different control from holding search budget fixed, and the excerpt reports no test-time-search baseline. The disjoint TerminalWorld task set does address that note's second confound, evolution and evaluation sharing one benchmark. The small transfer result, 84.2% on SWE-bench Verified from a Terminal-Bench 2.1 harness, "inside a narrow band" around the fix-skill reference, fits Do harness edits learn reusable strategies or memorize task fixes?, though the excerpt never inspects what the edits contain.

The excerpt does not establish which part matters. No ablation of the contract, the archive or the signal interface appears, so the claim that population selection beats single-lineage search rests on the paper's motivation rather than on a comparison shown here. It also does not say what the 17-point average is taken over, and it gives no run counts, variance, compute or search budget. "Audit-clean" is undefined, and transfer is measured in one direction only. The defensible reading is that a population-level search with a non-regression gate can lift a frozen-model agent on verifier-scored tasks, with in-domain gains far larger than the transferred one. It is not yet evidence that any single component is necessary.

Inquiring lines that read this note 10

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How does harness optimization generalize across different model architectures and domains?

Related concepts in this collection 3

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
13 direct connections · 65 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

selection over a population of harness variants under a preserve-and-extend contract improves a frozen-model agent without training new weights