INQUIRING LINE

Do the scaffolding, tools and prompts wrapped around an AI help average-strength models more than the very best ones?

Can mid-tier models benefit more from harness improvements than frontier models?

This explores whether the scaffolding built around a model (its prompts, tools, and control flow, called the harness) helps middle-capability models more than the strongest ones, and why that might be.


This asks whether the scaffolding around a model helps mid-tier models more than frontier ones. The corpus's most direct evidence says yes, with a twist. One study found that a model's ability to write good harness updates is flat across capability tiers. Its ability to benefit from those updates follows an inverted U that peaks in the middle: Do stronger models always evolve harnesses better?. Weak models often fail to invoke the harness at all. Strong models struggle with faithful instruction-following, so they drift from the prescribed steps. Being smarter doesn't mean using the scaffolding better.

The weak end isn't hopeless, though. It depends on how the harness is designed. A behavior-to-code map let weaker planners match stronger models at finding the relevant code, while using 8-13% fewer tokens: Can explicit behavior maps help weaker planners compete with stronger models?. In another case a stronger model built an inference-time harness that nearly doubled a weaker model's Theory-of-Mind score. It did this mostly by moving shaky reasoning into deterministic code and routing tasks, not by asking for longer thinking: Can a stronger model lift a weaker one at test time without retraining?. Both harnesses take decisions away from the model instead of asking it to make good ones. That suggests, as my reading and not a claim from the notes, that where the peak falls depends on how much judgment the harness demands of the model.

Frontier models still gain. A runbook tuned around frozen weights reached 95.3% on GPT-5.6 on Terminal-Bench 2.1 and lifted DeepSeek-V4 Flash by 5.4 points: Can execution harnesses lift model performance without retuning weights?. Evolving harness variants with the model frozen gave an average 17-point gain across benchmarks: Can frozen models improve by evolving their harnesses?. Neither result is split by model tier, so they can't say who gains most. A model already near 95% also has little room left to gain, and the notes don't test that directly.

Two cautions apply. First, harness gains have to be measured against a baseline that gets the same search and inference budget. Otherwise you may be crediting the harness for what extra computation would have bought anyway: How should we measure gains from automatic harness evolution?. Second, harness improvements seem to be a separate lever from model improvements. Automated optimization across many environments found mechanisms that cut token traffic by about half at comparable performance: Can agent harnesses be automatically optimized across many environments?. Routing queries among specialized models is another lever that can beat a single frontier model: Can routing beat building one better model?.

The corpus suggests mid-tier models are the sweet spot for harness gains. They are capable enough to follow the scaffolding but not so strong that they stop following it faithfully. The corpus has one direct study of this and no head-to-head comparison across tiers, so treat it as a strong hint and not a settled result.


Sources 8 notes

Do stronger models always evolve harnesses better?

Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.

Can explicit behavior maps help weaker planners compete with stronger models?

A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.

Can a stronger model lift a weaker one at test time without retraining?

A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.

Can execution harnesses lift model performance without retuning weights?

StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.

Can frozen models improve by evolving their harnesses?

DarwinX achieves average 17-point gains across benchmarks by evolving harness variants (prompts, tools, skills, control flow) under a preserve-and-extend contract while keeping the model frozen. Key evidence includes Terminal-Bench 2.1 rising to 84.7% and WebArena-Infinity reaching 93.0% audit-clean pass@1.

Show all 8 sources
How should we measure gains from automatic harness evolution?

Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.

Can agent harnesses be automatically optimized across many environments?

Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.

Can routing beat building one better model?

Avengers-Pro achieves 7% higher accuracy than GPT-5-medium by routing queries to optimal models per semantic cluster, or matches its performance at 27% lower cost. Ten 7B models with routing previously surpassed GPT-4.1 and 4.5, suggesting selection is a stronger lever than scaling.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.