AI performance gets credited to the model, but the prompts, tools, and code around it may deserve the credit or blame.
What makes a harness a first-class object rather than invisible scaffolding?
This explores what it means to treat a harness (the prompts, tools, memory, and code wrapped around a model) as something you measure, organize, and govern in its own right, rather than a background detail credited to the model.
This explores what it means to treat a harness (the prompts, tools, memory, and code wrapped around a model) as something you measure, organize, and govern in its own right. The corpus suggests three things turn it into an object: it has effects you can separate from the model's, it has a structure you can get lost in, and it holds the rules that keep an agent safe. When a harness stays invisible, credit and blame both go to the wrong place.
The first test is whether you can isolate what the harness contributes. Automatic harness evolution only shows real design value when it beats task-level test-time search given the same feedback and inference budget. Anything less could just be extra computation (How should we measure gains from automatic harness evolution?). The harness also interacts with the model in a way that's easy to miss. Models of every tier are about equally good at proposing useful harness edits, but the ability to benefit from them peaks in mid-tier models. Weak models fail to invoke the harness at all, and strong ones struggle to follow its instructions faithfully (Do stronger models always evolve harnesses better?). So the harness is not a neutral wrapper that lifts every model equally.
The second sign is that the harness has an internal structure that decides how well anyone can work with it. The hard part of evolving a production harness turns out to be finding every place a behavior lives, not writing the edit. A single behavior is spread across files, functions, and stages, while requests arrive in terms of behavior (Why is finding distributed behavior code so hard?). Reorganizing the repository around a behavior-to-code map raised win rates by 10–19 points and cut planner tokens. It also let weaker planners match stronger models at locating the right code (Can explicit behavior maps help weaker planners compete with stronger models?). Here a design choice about the harness did what a bigger model would otherwise have done. The same lens exposes a limit: most evolved edits memorize task-specific fixes an agent could rediscover in a single rollout, and few distill reusable strategies (Do harness edits learn reusable strategies or memorize task fixes?).
The third sign is that the harness carries the rules. One persistent agent logged 889 governance events over 96 active days, with its safeguards written into the memory layer it consulted while working. This worked better than an external policy because the agent actually read it (Can governance rules embedded in runtime memory actually protect autonomous agents?). Auditing follows the same logic. Scoped audit agents get a narrow remit, fixed artifacts, and a duty to cite evidence. Infrastructure records become the checks nobody can argue with, and the agent's judgment is the arguable step layered on top (Can scoped agents reliably judge semantic hacks in runtime analysis?).
The cost of leaving all this invisible shows up in one study. It bundled clear authorization rules with restricted tools and reported zero protected-test modifications. But it never ran a single-factor ablation, so no one can say whether the tools were unavailable or the agent simply chose not to use them (Do authorization rules or restricted tools prevent test modifications?). If you can't take a harness apart and test its pieces, you can't say what is protecting you, and you can't say what is improving your results.
Sources 8 notes
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
The core difficulty in evolving production harnesses is not generating edits but finding every code location that implements a behavior. Harnesses distribute single behaviors across files, functions, and stages, creating a representational mismatch between behavioral requests and structural code organization.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
Analysis of evolved harness trajectories shows rational, well-motivated edits across prompt and tool layers, but most persist fixes an agent could rediscover in a single rollout. Gains remain limited because memorized shortcuts cache what's already within reach rather than converting hard failures into successes.
Show all 8 sources
A persistent agent recorded 889 governance events across 96 active days, with safeguards encoded directly into the memory layer the agent consulted during operation. Runtime-resident governance proved more effective than external policies because the agent actually accessed it during decision-making.
BenchShield constrains audit agents by limiting their remit, fixing the artifacts they see, and requiring evidence citation. This positions infrastructure records as unchallengeable checks and audit judgments as the arguable step after them, though reported reliability remains unquantified.
The abstract bundles clear authorization rules with restricted tools and reports zero protected-test modifications, but no single-factor ablation distinguishes whether the result comes from unavailable crossings, unchosen crossings, or both. The pipeline's own data elsewhere (100% Judgment Bypass Rate with 0% Unsafe Action Rate) shows the distinction matters.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Rethinking the Evaluation of Harness Evolution for Agents
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
- ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement