INQUIRING LINE

Upgrading the scaffolding around an AI model helps most in the middle: weak models ignore it, strong ones bend its rules.

Which foundation model tiers most benefit from harness updates?

This explores which kinds of models (weak, mid-tier, or frontier) actually get the biggest lift when their surrounding harness (prompts, tools, control flow) is updated, rather than which models are best at writing those updates.


This explores which kinds of models (weak, mid-tier, or frontier) actually get the biggest lift when their surrounding harness (prompts, tools, control flow) is updated, rather than which models are best at writing those updates. The corpus gives a surprisingly specific answer: the benefit peaks in the middle. One study finds that the ability to *produce* useful harness edits is flat across model tiers, but the ability to *benefit* from those edits follows an inverted U. Weak models often fail to invoke the harness at all, while strong models struggle with faithfully following the instructions it imposes (Do stronger models always evolve harnesses better?). So a harness is only as good as the model's willingness and ability to actually use it.

That inverted U is a single finding, but other work points the same way at the component level. A controlled coding study found that no harness part is universally valuable: planning helps weaker models get the job done, but for stronger models it mostly cuts cost, and context management pays off most when the window is tight (Which coding harness components matter most in different conditions?). Safety harnesses show the mirror-image problem. A harness strict enough for one model over-blocks another, so model characteristics help decide which harness works (Should safety harnesses be customized for each deployment?). In both cases the answer to "who benefits?" depends on the pairing of harness and model.

There is also a counterweight to the mid-tier story. A stronger model can build inference-time harnesses that nearly double a weaker model's score on Theory-of-Mind benchmarks. It does this mostly by moving unstable reasoning into deterministic code and task-specific routing (Can a stronger model lift a weaker one at test time without retraining?). That suggests weak models can benefit a lot when the harness does the work for them, rather than asking them to follow subtle instructions. This sits in tension with the finding that weak models fail to invoke harnesses, so the design of the harness (code that runs regardless, versus advice the model must choose to use) may matter as much as the tier.

At the top end, harness gains don't disappear. A runbook optimized around frozen weights lifted several models on Terminal-Bench 2.1, reaching 95.3% on GPT-5.6, and transferred to newer models without modification (Can execution harnesses lift model performance without retuning weights?). Evolving harness variants under a preserve-and-extend contract gave about 17-point average gains on a frozen model (Can frozen models improve by evolving their harnesses?). These results don't split cleanly by tier, and the corpus doesn't offer a head-to-head comparison across weak, mid, and frontier models. One caution applies to all of them: harness-evolution gains need to be measured against matched-budget test-time search, or a gain may just reflect extra computation (How should we measure gains from automatic harness evolution?).

The best-supported answer is mid-tier models, with real caveats. The inverted-U result comes from one study, and the other notes show that the benefit depends on which harness component you change, how tight the context is, and whether the harness is enforced in code or left as advice. If you're choosing where to invest in harness work, the mid-tier sweet spot is a reasonable default, but the corpus suggests testing it on your own model and task.


Sources 7 notes

Do stronger models always evolve harnesses better?

Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.

Which coding harness components matter most in different conditions?

A controlled study varying planning, action space, and context management across models and budgets found that context management becomes most valuable under tight windows, while planning shifts from helping weaker models to cutting costs for stronger ones.

Should safety harnesses be customized for each deployment?

A harness strict enough for one model over-blocks another, while policies general enough to transfer across domains miss application-specific safety relations. Domain semantics and model characteristics jointly determine which harness is effective.

Can a stronger model lift a weaker one at test time without retraining?

A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.

Can execution harnesses lift model performance without retuning weights?

StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.

Show all 7 sources
Can frozen models improve by evolving their harnesses?

DarwinX achieves average 17-point gains across benchmarks by evolving harness variants (prompts, tools, skills, control flow) under a preserve-and-extend contract while keeping the model frozen. Key evidence includes Terminal-Bench 2.1 rising to 84.7% and WebArena-Infinity reaching 93.0% audit-clean pass@1.

How should we measure gains from automatic harness evolution?

Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.