Can a smarter wrapper around an AI — better memory, tools, and routing — fix things that simply building a bigger model never could?
Can better agent harnesses solve problems that model scale alone cannot?
This explores whether the scaffolding around a model (its memory, tools, routing, and how it runs actions) can fix problems that simply training bigger models doesn't, or whether harness gains are just a cheaper stand-in for scale.
This explores whether the system built around a model, its harness, can solve problems that a bigger model alone can't, or only saves money on the way to the same result. The corpus answers yes, with conditions. The clearest evidence is that harness work lifts models whose weights stay frozen. One execution-harness runbook raised Terminal-Bench 2.1 scores across several models and carried over to newer models without changes Can execution harnesses lift model performance without retuning weights?. Automated harness optimization across many environments found four mechanisms that cut token traffic by nearly half without losing performance Can agent harnesses be automatically optimized across many environments?. In that work, harness gains look orthogonal to model gains: they stack on top of scale rather than repeating it.
The stronger claim is that some problems sit outside what scale can reach. One line of work argues that agent reliability comes from moving three burdens out of the model: memory, reusable skills, and interaction protocols. A model that holds all of these internally has to solve them again on every task Where does agent reliability actually come from?. Another finds a ceiling that no single agent loop can get past. Work that needs different kinds of expertise, parallel execution, and independent checking is an organizational problem, not a capability problem Do single agents always hit organizational limits?. Better structure can also stand in for raw capability. Reorganizing a codebase around how it behaves at runtime let a weaker planner match stronger models at finding the right code to change Can explicit behavior maps help weaker planners compete with stronger models?. Taken together, these suggest that cheap small models can handle most agent subtasks, with large models called only when needed Can small language models handle most agent tasks?.
Two findings complicate the simple story that a better harness is always better. First, the benefit isn't spread evenly across model sizes. Models at every tier are about equally good at proposing useful harness edits, but how much a model gains from them peaks in the middle. Weak models fail to use the harness at all, and the strongest models are worse at following its instructions faithfully Do stronger models always evolve harnesses better?. Second, the usual way of making models better, post-training, can work against harnesses. Base models with only a loose system prompt eventually find more distinct solutions than their post-trained versions as you give them more attempts. Post-training sharpens performance on easy cases and removes rare but reachable solutions Do base models find more solutions than post-trained ones?. So training and scaffolding don't simply add up. Sometimes a looser model in a good harness explores more widely.
The framing of harness versus scale may also be too sharp, because the two feed each other. Surveys of self-improving agents describe a slow loop that updates the model's weights and a fast loop that updates prompts, memory, and tools. Most recent progress happens in the fast loop because those changes are cheap and easy to undo Do self-improving agents really split into two distinct loops?. A deployed routing harness also records which tasks were hard and how they turned out. That data can be converted into training examples for the next model Can a routing harness generate its own training data automatically?. In that setup the harness is also a source of training data.
The real limit for both may be data. Agents trained on expert demonstrations never learn beyond what the people who built the dataset imagined Can agents learn beyond what their training data shows?. One proposed escape is a language model trained to simulate agent environments, which gives agents places to practice that don't exist in the real data Can language models learn to simulate agent environments?. So the question underneath this one may be less about harness versus scale and more about what either one gets to learn from.
Sources 12 notes
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
Research shows that real-world tasks requiring heterogeneous expertise, parallel execution, and independent verification exceed what any single agent loop can organize. Graph-based system abstractions are needed to distribute intelligence across specialized agents.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
Show all 12 sources
SLMs handle the repetitive, well-defined language tasks that constitute most agent work at 10–30× lower cost than LLMs, making heterogeneous architectures (SLMs by default, LLMs selective) the economically rational design pattern.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Across 14 model pairs and three agentic benchmarks, base models equipped with only relaxed system prompts eventually surpass post-trained counterparts in pass@K coverage as rollout budget grows. Post-training bimodalizes task outcomes, sharpening performance on easy cases while eliminating rare-but-reachable solutions.
A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.
A deployed routing system records execution trajectories, capability demand estimates, and outcome data that can be converted into labeled training examples for fine-tuning and distillation, turning the harness into both a serving component and a difficulty labeler.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
Qwen-AgentWorld demonstrates that native language world models trained via next-state prediction on 10M+ trajectories outperform real-environment training on three benchmarks and transfer across seven domains, positioning next-state prediction as a foundation objective for agents.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
- Sharpening Tax in Post-Training
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- Rethinking the Evaluation of Harness Evolution for Agents
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses