When an AI agent gets things done, how much credit goes to the model and how much to its surrounding tools?
How much realized agent capability comes from the harness versus the model?
This explores how to split credit for what an agent can actually do between the harness (the memory, tools, skills and execution loop wrapped around a model) and the model's own weights.
This explores how to split credit for what an agent can actually do between the harness around a model and the model itself. The corpus has no clean percentage split, and its evidence suggests the question is slightly malformed, because the two contributions interact. But it does show the harness is far bigger than most people assume.
The strongest case for the harness is that gains show up with the model frozen. One system optimized the execution environment around fixed weights and lifted Terminal-Bench 2.1 accuracy across several models. The same runbook carried over to newer models unchanged, reaching 95.3% on GPT-5.6 and adding 5.4 points to DeepSeek-V4 Flash (Can execution harnesses lift model performance without retuning weights?). Reorganizing a code repository around runtime behavior, rather than file layout, let a weaker planner match stronger models at finding the right code, with win rates up 10–19 points (Can explicit behavior maps help weaker planners compete with stronger models?). Automated search across many environments found four harness mechanisms that cut token traffic by 44.7–49.0% at comparable performance. That is an efficiency gain rather than a capability gain, but the authors read it as harness improvement running orthogonal to model improvement (Can agent harnesses be automatically optimized across many environments?). The framing behind this is that reliability comes from moving memory, skills and interaction protocols out of the model and into the harness, so the model doesn't have to re-solve the same problems on every run (Where does agent reliability actually come from?, Where does agent reliability actually come from?).
The model side still matters, though, and it matters unevenly. Models of every tier are about equally good at writing useful harness edits, but the ability to benefit from those edits peaks in mid-tier models. Weak models fail to invoke the harness at all, and strong models struggle to follow its instructions faithfully (Do stronger models always evolve harnesses better?). A harness is therefore not a plug-in that adds a fixed amount of capability. Its value depends on a model that can use it. Related work finds that the same harness performs very differently under different executors, and that building and maintaining a harness is a separate, measurable skill that downstream task scores don't reveal (Can language models build and maintain their own agent harnesses?). Some capabilities also stay firmly in the weights. In ultra-long optimization tasks, the best predictor of success was persistence in the benchmark, edit and incorporate loop rather than first-attempt quality, and most of the 17 frontier models quit early or burned their budget (What predicts success in ultra-long-horizon agent tasks?).
The boundary is also moving. Some of what the harness supplies is being trained back into models: a 35B model trained on execution-grounded data and long-horizon trajectories reaches the cost-performance frontier against much larger models, so training for follow-through can substitute for raw scale (Does model efficiency matter more than peak capability for real work?). Meanwhile, self-improving agents split into a slow loop that updates weights and a fast loop that updates prompts, memory and tools. Recent progress clusters in the fast loop, partly because scaffold changes are cheaper and reversible (Do self-improving agents really split into two distinct loops?). So harness gains look large in part because they are easy to iterate on, not because they are inherently larger than weight gains.
The corpus also suggests why no one can give you a number. Identical success rates can hide huge differences in efficiency, reliability and memory hygiene, which is exactly the layer where harness and model contributions get tangled (How should we measure agent system performance beyond task success?). And benchmark wins on contest-style tasks don't predict performance on long-horizon professional work (Why do agent benchmarks not predict real economic value?). Any attribution figure inherits those blind spots. The most useful conclusion the notes support is that the harness can carry a lot of realized capability, but only up to what the model can absorb. For a mid-tier model that ceiling is high.
Sources 12 notes
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
Applied AI research shows capability shifts from model weights to external structures like memory and skills. However, reusable skills bundle executable code and system reach, creating security costs that traditional lifecycle inspection cannot catch when attacks compose across multiple skills.
Show all 12 sources
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Research shows that LLMs vary sharply in building harnesses across domains, struggle to retain useful intermediate updates during evolution, and produce harnesses whose performance shifts dramatically with different executors—demonstrating that harness quality cannot be inferred from downstream task scores alone.
Across 17 frontier models on 36 expert-curated optimization tasks, repeated benchmark-edit-incorporate cycles within a wall-clock budget proved the dominant success predictor. Most models terminated early or burned budget unproductively; Claude Opus 4.6 stood out as persistent.
Occamy-1.0, a 35B-parameter model further trained on execution-grounded data and long-horizon trajectories, achieves competitive performance with much larger models while sitting at the low-cost knee of the Pareto frontier, suggesting that training for coordination and follow-through substitutes for raw scale in multi-step work.
A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.
Single task-success metrics obscure how agents achieve results across memory, context, and verification layers. Research shows identical success rates can mask enormous differences in efficiency, reliability, and deployment readiness—requiring harness-level benchmarks that measure trajectory, memory hygiene, and verification costs.
ALE's analysis of 960 real occupational workflows shows agents excel at abstract contests but fail long-horizon professional tasks. The gap is not model capability but benchmark design—the field optimizes what it measures, and it has measured contests rather than work.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable