Upgrading an AI agent's scaffolding helps mid-tier models most — the weakest can't use it, and the strongest ignore it.
Can harness updates benefit agents equally across all model sizes?
This explores whether upgrades to an agent's harness — its prompts, tools, and memory scaffolding — help every model equally, or whether some sizes of model gain more than others.
This explores whether upgrades to an agent's harness — its prompts, tools, and memory scaffolding — help every model equally. The corpus says no, and in a surprising way: the ability to *write* a good harness update is roughly flat across model sizes, but the ability to *benefit* from one follows an inverted-U curve that peaks at mid-tier models Do stronger models always evolve harnesses better?. Weak models can't reliably invoke the harness even when handed a good one; the strongest models struggle to follow the harness's instructions faithfully, often overriding them. So the same edit lands unevenly, and the middle of the range is where it pays off most.
That asymmetry makes more sense once you see the harness as a place to *offload cognitive work* rather than a productivity multiplier applied uniformly. Reliable agents externalize three burdens — memory, skills, and interaction protocols — into a harness layer so the model doesn't re-solve the same problems every run Where does agent reliability actually come from?. A weak model lacks the baseline competence to use those externalized structures; a very strong model already carries much of that competence internally and gains less from the scaffold. This is also why context management helps different agents differently: an external manager should preserve high-fidelity context for strong agents but aggressively compress for weak ones to keep them reliable Can external managers compress context better than frozen agents?.
The most counterintuitive part is that a good harness can *invert* the usual size advantage. Reorganizing a harness repository around runtime behavior let a weaker planner match stronger models' code localization across every precision and recall metric, while cutting tokens Can explicit behavior maps help weaker planners compete with stronger models?. So harness updates aren't just unequal — they can be a substitute for scale, closing gaps that raw model size would otherwise open. That reframes the whole cost question: if small models handle most agentic subtasks at a fraction of the price Can small language models handle most agent tasks?, then investing in the harness rather than the model is often the economically rational move.
There's a catch worth knowing before you assume any harness update is a win. Most evolved harness edits turn out to memorize task-specific fixes rather than distill reusable strategies — they cache shortcuts for already-solvable problems instead of converting failures into successes Do harness edits learn strategies or just memorize fixes?. And because harness evolution is itself a search loop, its reported gains are confounded with sheer search effort; only improvements beyond an equal-budget baseline can be credited to the design Are harness evolution gains really from better design?. So the honest answer is layered: harness updates don't benefit all sizes equally, they benefit mid-tier models most, they can let small models punch above their weight — but only if the update genuinely encodes strategy rather than memorized answers.
Zooming out, this sits inside a broader split in how agents improve at all: a slow parametric loop that rewrites model weights versus a fast non-parametric loop that rewrites prompts, memory, and tools Do self-improving agents really split into two distinct loops?. Harness updates are the fast loop, and its appeal is precisely that scaffold edits are cheap and reversible where weight updates are not. Related work shows agents can even adapt continuously through memory alone, with no parameter changes at all Can agents learn continuously from experience without updating weights? — which is really the same bet as the harness question, just taken to its limit.
Sources 9 notes
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
An external RL-trained manager can adaptively prune context for frozen agents, with the key insight that stronger agents benefit from high-fidelity preservation while weaker agents need aggressive compression to stay reliable.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
SLMs handle the repetitive, well-defined language tasks that constitute most agent work at 10–30× lower cost than LLMs, making heterogeneous architectures (SLMs by default, LLMs selective) the economically rational design pattern.
Show all 9 sources
Inspecting evolved harness edits reveals rational changes across prompt and tool layers, but most persist fixes the agent could rediscover in one rollout. Gains remain limited because edits cache shortcuts for already-solvable tasks rather than converting failures into successes.
Harness evolution is itself a search loop, so its reported improvements are confounded with search effort. Only improvements beyond equal task-level search budget can be attributed to harness design, and held-out evaluation is needed to rule out task memorization.
A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
- Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
- Large Language Model Agents Are Not Always Faithful Self-Evolvers
- Useful Memories Become Faulty When Continuously Updated by LLMs