Wrapping an AI in tools and instructions helps most in the middle of the pack — why do top models gain less?
Why does harness benefit capacity peak at mid-tier models, not frontier scale?
This explores why the ability to benefit from a harness (the scaffolding of instructions, tools, and code organization wrapped around a model) is strongest in middle-capability models and not in the strongest ones.
This explores why the ability to benefit from a harness (the scaffolding of instructions, tools, and code organization wrapped around a model) is strongest in middle-capability models and not in the strongest ones. The corpus splits the question into two abilities that people usually lump together. Writing useful harness edits stays about the same across model tiers. Benefiting from those edits follows an inverted U, peaking in the middle Do stronger models always evolve harnesses better?. A stronger model doesn't automatically evolve or use a harness better.
The peak sits where two different failures cross. Weak models fail on the near side because they don't reliably invoke the harness at all, so the scaffolding goes unused. Strong models fail on the far side because they struggle with faithful instruction-following Do stronger models always evolve harnesses better?. A harness only helps if the model actually does what it says. The note gives the outcome but not a detailed mechanism for the strong-model side. One reading is that a very capable model is more inclined to substitute its own judgment for the rules it was handed. Mid-tier models are capable enough to use the structure and still compliant enough to follow it.
A second result from the corpus explains why the middle is where a harness pays off most. A harness is most valuable where it fills a gap the model can't close alone. Reorganizing a code repository around runtime behavior let weaker planners match stronger models' code localization on every precision and recall metric. It also raised win rates by 10–19 points while using 8–13% fewer planner tokens Can explicit behavior maps help weaker planners compete with stronger models?. Structure can stand in for raw capability, but only for a model that can make use of it.
This fits a wider pattern in the collection: parameter count is one lever among several. Smaller models given more inference-time compute can match larger ones on hard prompts Can inference compute replace scaling up model size?. Routing each query to the right specialist model has beaten single frontier models, at lower cost or with higher accuracy Can routing beat building one better model?. When you read any "harness helps" result, ask whether the gain beats a task-level test-time search given the same feedback and inference budget. Only the part beyond that baseline can be credited to the harness design and not to extra computation How should we measure gains from automatic harness evolution?.
Sources 5 notes
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
Snell et al. (2024) showed that inference-time compute trades off against model parameter scaling, especially on difficult prompts. This reveals pretraining and inference compute are not independent resources.
Avengers-Pro achieves 7% higher accuracy than GPT-5-medium by routing queries to optimal models per semantic cluster, or matches its performance at 27% lower cost. Ten 7B models with routing previously surpassed GPT-4.1 and 4.5, suggesting selection is a stronger lever than scaling.
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
- Scaling Laws for Agent Harnesses via Effective Feedback Compute