Can language models build and maintain their own agent harnesses?

The paper's claim is that whether a model can create and maintain its own agent harness is a separate question from whether it can solve tasks inside a harness someone else built, and that current evaluations skip it. The abstract notes that "changing this harness while holding model weights fixed can substantially alter task performance," yet agent evaluations "typically report downstream performance under a chosen harness." HarnessDev instead makes "runnable infrastructure" the unit of evaluation. In Creation, an agent starts from a minimal seed and a few cases and builds a complete execution system. In Evolution, it revises its own harness from downstream execution feedback. Each harness is then scored on held-out capability and on execution-token cost.

The discussion reports three results that support the separation. First, creation quality varies sharply by domain: under Self-Eval, models match the human reference in writing, exceed it in machine-learning experimentation, remain far behind in search and research, and still trail in code. Second, cross-executor comparisons show some harnesses improving under a stronger executor while others "exhibit creator co-adaptation." Third, evolution is harder than creation: "useful intermediate updates are often erased by later changes, and more updates do not guarantee a positive final gain." The fixed-Gemini ablation adds that changing only the runtime binding "can substantially move H0 and alter which harness changes are useful." The introduction supplies the motivating gap, taken from a cited source: GPT-5 solves 35.2% of Terminal-Bench 2.1 inside Terminus 2 and 49.6% inside Codex CLI.

This extends the library's harness-evolution notes from a new direction. Do stronger models always evolve harnesses better? splits self-evolution into updating and benefiting; HarnessDev adds the creation stage, which starts from a seed rather than an existing harness, and the executor swap, which shows the same harness behaving differently under different runtimes. Do harness edits learn reusable strategies or memorize task fixes? diagnoses what evolution edits contain; the erasure of useful intermediate updates is a different observation, about whether gains survive later revisions. It is not shown to share that note's cause. The paper also treats as an object of measurement what What are the three distinct layers of agent code? treats as a given: the middle layer, the harness, is here something the model itself must produce.

The excerpt leaves most of the evidence unstated. It does not name the six creator LLMs, give per-domain scores or gap sizes, define the human reference or Self-Eval beyond their names, explain what H0 is, or report Evolution sample sizes; the 2,207 downstream instances are stated for Creation only. It does not test why intermediate updates are erased or what "co-adaptation" consists of. What it supports is directional: a score obtained under a fixed harness cannot stand in for a model's ability to build or maintain one, and a harness cannot be judged apart from the executor it runs under.