Do stronger models always evolve harnesses better?
We explore whether base model capability predicts both the ability to write useful harness updates and the ability to benefit from them. The answer reshapes how we should allocate capability in self-evolving agent systems.
Self-evolving agents edit an external harness — prompts, skills, memories, tools — from execution evidence, without touching model parameters. The natural assumption is that stronger base models do this better on both ends. This paper disentangles two distinct capabilities and finds neither follows that assumption.
Harness-updating — producing persistent edits that lead to gains — is flat in base capability. Models across capability tiers produce updates yielding surprisingly similar gains; even a Qwen3.5-9B evolver induces gains comparable to Claude Opus 4.6. Writing a good skill or memory is apparently not bottlenecked by raw model strength.
Harness-benefit — actually improving when handed an updated harness — is non-monotonic. Weak-tier models gain little, mid-tier models benefit most, and strong-tier models benefit less than mid-tier. Two failure modes explain the weak end: failing to activate the relevant harness artifact, and failing to follow it faithfully once activated.
The practical inversion is sharp: invest capability budget in the agent that uses the harness, not the evolver that writes it — and target agent training at harness invocation and long-horizon instruction-following rather than at generating cleverer updates. This complicates the "let a frontier model improve everything" intuition and connects to Why do better reasoning models ignore instructions?: strong models may benefit less precisely because the bottleneck is faithful instruction-following, which scaling erodes.
Inquiring lines that read this note 34
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do harness improvements transfer across model scales or memorize shortcuts?- Can mid-tier models benefit more from self-generated harness updates than others?
- Can smaller models produce skill updates as useful as frontier model updates?
- What happens when different harnesses project the same model?
- Does harness benefit depend on which model tier you use?
- What cognitive burdens should move from model parameters into harness infrastructure?
- What causes weak models to fail at activating harness artifacts?
- How should we allocate model budget between evolvers and harness users?
- Can harness updates benefit agents equally across all model sizes?
- How should harness scaffolding be treated as a first-class object?
- Why do mid-tier models benefit more from memorized harness shortcuts?
- Can harness evolution be redirected toward distilling transferable procedures instead?
- Why do evolved harness edits mostly memorize rather than generalize?
- What feedback signals matter most during harness evolution search?
- Should we train the evolver or the executor when building self-improving agents?
- How can agents evolve their own skills without human input?
- Can agent-authored skill libraries compound autonomy gains over time?
- What stops evolved agent behaviors from generalizing beyond specific tasks?
- Why do self-improving agents concentrate progress in the fast non-parametric loop?
- Can skill repositories evolve toward execution-oriented refinement over time?
- What makes skills worth externalizing into a persistent harness?
- Why does the harness layer accumulate distributed behaviors over time?
- What persistent failures remain unsolved despite harness evolution efforts?
- How should versioning and rollback govern the fast scaffold update loop?
- What makes behavior localization the bottleneck in agent harness evolution?
- Why do persistent, resynchronized artifacts compound harness capability gains?
- How do agent-created code artifacts become part of harness infrastructure?
- What makes agent-initiated artifacts the underexplored frontier in harness engineering?
- Are durable shared code artifacts better than per-task harness patches?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Where does agent reliability actually come from?
Exploring whether LLM agent performance depends on larger models or on thoughtful system design choices like memory, skills, and protocols that shift cognitive work outside the model.
sharpens it: which model should hold which role in a harness-evolution loop
-
Can a separate trained curator improve skill libraries better than frozen agents?
Explores whether decoupling skill curation from agent execution enables better long-term learning of what skills to keep, delete, or refine. Matters because manual curation doesn't scale and heuristic approaches lack feedback.
same updater/executor split; this note adds the capability-tier asymmetry
-
How can agent self-evolution be made safe and auditable?
As agents begin updating their own prompts and tools, how can we track these changes, measure their effects, and safely reverse problematic updates? This matters because untracked evolution leads to unmaintainable systems and makes regressions impossible to diagnose.
the protocol substrate this capability analysis runs on top of
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- Code as Agent Harness
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
- Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
- Self-Improvements in Modern Agentic Systems: A Survey
Original note title
the capacity to produce useful harness updates is flat across model tiers but the capacity to benefit from them peaks at mid-tier