Can language models build and maintain their own agent harnesses?
This explores whether an LLM's ability to create and revise its own execution infrastructure is a distinct skill from solving tasks within someone else's harness, and whether current evaluations overlook this capability.
The paper's claim is that whether a model can create and maintain its own agent harness is a separate question from whether it can solve tasks inside a harness someone else built, and that current evaluations skip it. The abstract notes that "changing this harness while holding model weights fixed can substantially alter task performance," yet agent evaluations "typically report downstream performance under a chosen harness." HarnessDev instead makes "runnable infrastructure" the unit of evaluation. In Creation, an agent starts from a minimal seed and a few cases and builds a complete execution system. In Evolution, it revises its own harness from downstream execution feedback. Each harness is then scored on held-out capability and on execution-token cost.
The discussion reports three results that support the separation. First, creation quality varies sharply by domain: under Self-Eval, models match the human reference in writing, exceed it in machine-learning experimentation, remain far behind in search and research, and still trail in code. Second, cross-executor comparisons show some harnesses improving under a stronger executor while others "exhibit creator co-adaptation." Third, evolution is harder than creation: "useful intermediate updates are often erased by later changes, and more updates do not guarantee a positive final gain." The fixed-Gemini ablation adds that changing only the runtime binding "can substantially move H0 and alter which harness changes are useful." The introduction supplies the motivating gap, taken from a cited source: GPT-5 solves 35.2% of Terminal-Bench 2.1 inside Terminus 2 and 49.6% inside Codex CLI.
This extends the library's harness-evolution notes from a new direction. Do stronger models always evolve harnesses better? splits self-evolution into updating and benefiting; HarnessDev adds the creation stage, which starts from a seed rather than an existing harness, and the executor swap, which shows the same harness behaving differently under different runtimes. Do harness edits learn reusable strategies or memorize task fixes? diagnoses what evolution edits contain; the erasure of useful intermediate updates is a different observation, about whether gains survive later revisions. It is not shown to share that note's cause. The paper also treats as an object of measurement what What are the three distinct layers of agent code? treats as a given: the middle layer, the harness, is here something the model itself must produce.
The excerpt leaves most of the evidence unstated. It does not name the six creator LLMs, give per-domain scores or gap sizes, define the human reference or Self-Eval beyond their names, explain what H0 is, or report Evolution sample sizes; the 2,207 downstream instances are stated for Creation only. It does not test why intermediate updates are erased or what "co-adaptation" consists of. What it supports is directional: a score obtained under a fixed harness cannot stand in for a model's ability to build or maintain one, and a harness cannot be judged apart from the executor it runs under.
Inquiring lines that read this note 25
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do harness improvements deliver capability gains comparable to model scaling?- Can runtime behavior mapping help localize harness deficiencies?
- What should an external contract for model improvement actually contain?
- Which domains see models exceed human harness design quality?
- How much does executor choice change a harness's actual performance?
- Do models co-adapt their harnesses to specific executor strengths?
- Can harness edits distill reusable strategies or mostly memorize task-specific fixes?
- How much realized agent capability comes from the harness versus the model?
- Can harnesses that rewrite themselves through reviewed commits achieve state-of-the-art performance?
- What makes a distilled skill verifiable and ready for agent execution?
- Why did older robot scientists log auditable provenance while LLM agents inherited only generation capability?
- How do humans handle verification scope when delegating creation to language models?
- Why might oversight focused only on language model outputs miss the agentic layer?
- Why do LLMs lack the internal commitment structures that BDI agents provide?
- How can formal communication layers be added to LLM agents without sacrificing flexibility?
- Can rules engines and LLM agents be compared by performance metrics?
- What five recurring pathways extend single-user LLM assistants into multi-party operation?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do stronger models always evolve harnesses better?
We explore whether base model capability predicts both the ability to write useful harness updates and the ability to benefit from them. The answer reshapes how we should allocate capability in self-evolving agent systems.
adds the from-seed Creation stage and the executor swap to the updater versus beneficiary split
-
Do harness edits learn reusable strategies or memorize task fixes?
When meta-agents evolve harnesses iteratively, do the persisted edits encode transferable procedures that solve new problems, or do they mostly cache shortcuts for already-solvable tasks? This matters because it determines whether harness evolution genuinely expands capability.
a different evolution failure: useful updates erased by later edits, not shown to be the same cause
-
What are the three distinct layers of agent code?
Does separating agent code into model capabilities, system harness, and agent-created artifacts help explain why agentic systems fail and where to intervene for improvement?
HarnessDev makes the system-provided middle layer something the model must itself produce
-
How should we measure agent system performance beyond task success?
Current evaluation metrics collapse agent behavior into a single success score, hiding critical information about how agents operate. What dimensions—trajectory quality, memory use, context efficiency, verification cost—should benchmarks actually measure?
same push away from task-output scoring; here the unit of evaluation is the runnable harness
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Rethinking the Evaluation of Harness Evolution for Agents
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- An Empirical Study of Harness Design for Coding Agents
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Original note title
harness development should be evaluated directly because the harness a model builds is separable from the model running inside it