INQUIRING LINE

Wrapping an AI in tools and instructions helps most in the middle of the pack — why do top models gain less?

Why does harness benefit capacity peak at mid-tier models, not frontier scale?

This explores why the ability to benefit from a harness (the scaffolding of instructions, tools, and code organization wrapped around a model) is strongest in middle-capability models and not in the strongest ones.


This explores why the ability to benefit from a harness (the scaffolding of instructions, tools, and code organization wrapped around a model) is strongest in middle-capability models and not in the strongest ones. The corpus splits the question into two abilities that people usually lump together. Writing useful harness edits stays about the same across model tiers. Benefiting from those edits follows an inverted U, peaking in the middle Do stronger models always evolve harnesses better?. A stronger model doesn't automatically evolve or use a harness better.

The peak sits where two different failures cross. Weak models fail on the near side because they don't reliably invoke the harness at all, so the scaffolding goes unused. Strong models fail on the far side because they struggle with faithful instruction-following Do stronger models always evolve harnesses better?. A harness only helps if the model actually does what it says. The note gives the outcome but not a detailed mechanism for the strong-model side. One reading is that a very capable model is more inclined to substitute its own judgment for the rules it was handed. Mid-tier models are capable enough to use the structure and still compliant enough to follow it.

A second result from the corpus explains why the middle is where a harness pays off most. A harness is most valuable where it fills a gap the model can't close alone. Reorganizing a code repository around runtime behavior let weaker planners match stronger models' code localization on every precision and recall metric. It also raised win rates by 10–19 points while using 8–13% fewer planner tokens Can explicit behavior maps help weaker planners compete with stronger models?. Structure can stand in for raw capability, but only for a model that can make use of it.

This fits a wider pattern in the collection: parameter count is one lever among several. Smaller models given more inference-time compute can match larger ones on hard prompts Can inference compute replace scaling up model size?. Routing each query to the right specialist model has beaten single frontier models, at lower cost or with higher accuracy Can routing beat building one better model?. When you read any "harness helps" result, ask whether the gain beats a task-level test-time search given the same feedback and inference budget. Only the part beyond that baseline can be credited to the harness design and not to extra computation How should we measure gains from automatic harness evolution?.


Sources 5 notes

Do stronger models always evolve harnesses better?

Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.

Can explicit behavior maps help weaker planners compete with stronger models?

A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.

Can inference compute replace scaling up model size?

Snell et al. (2024) showed that inference-time compute trades off against model parameter scaling, especially on difficult prompts. This reveals pretraining and inference compute are not independent resources.

Can routing beat building one better model?

Avengers-Pro achieves 7% higher accuracy than GPT-5-medium by routing queries to optimal models per semantic cluster, or matches its performance at 27% lower cost. Ten 7B models with routing previously surpassed GPT-4.1 and 4.5, suggesting selection is a stronger lever than scaling.

How should we measure gains from automatic harness evolution?

Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.