When an AI tops a leaderboard, how much credit belongs to the model and how much to the software wrapped around it?
How much does harness design contribute to reported model capability scores?
This explores how much of a benchmark score comes from the scaffolding around a model (prompts, tools, context handling, control flow) rather than from the model's weights.
This explores how much of a reported capability score comes from the harness around a model rather than the model itself. The corpus suggests it can be a lot, but no note gives a single share, and the size varies by task and model. The clearest evidence comes from freezing the weights and changing only the harness. Evolving harness variants (prompts, tools, skills, control flow) around a frozen model gave an average 17-point gain across benchmarks, with Terminal-Bench 2.1 rising to 84.7% (Can frozen models improve by evolving their harnesses?). Optimizing the execution system alone lifted DeepSeek-V4 Flash by 5.4 points, and the same runbook carried over to newer models unchanged, reaching 95.3% on GPT-5.6 (Can execution harnesses lift model performance without retuning weights?). A leaderboard number is therefore really a model-plus-harness number.
The harness can also stand in for model capability. A stronger model built inference-time harnesses that nearly doubled a weaker model's Theory-of-Mind scores without retraining. Most of the gain came from moving unstable reasoning into deterministic code and routing tasks, not from making the model think longer (Can a stronger model lift a weaker one at test time without retraining?). In code localization, giving a weaker planner a map from runtime behavior to code let it match stronger models on every precision and recall metric (Can explicit behavior maps help weaker planners compete with stronger models?). Some of the gap between two models on a leaderboard may be a harness gap.
The contribution isn't a constant, though. Context management pays off most when the context window is tight. Planning helps weaker models in some settings and mainly cuts costs for stronger ones (Which coding harness components matter most in different conditions?). Benefit from harness edits is also uneven across model tiers. Models of every strength can write useful harness updates about equally well, but mid-tier models benefit most. Weak models fail to invoke the harness, and strong ones struggle to follow it faithfully (Do stronger models always evolve harnesses better?). The same holds for safety harnesses: one strict enough for a model in one deployment over-blocks another model, so the fit between harness, model and domain matters (Should safety harnesses be customized for each deployment?).
The harder question is how much of a reported gain is harness design and how much is extra computation or fit to the benchmark. One note argues that a harness gain only counts to the extent it beats task-level test-time search given equal feedback and inference budget (How should we measure gains from automatic harness evolution?). Another evolves harness modules on data separate from the benchmark, to isolate reusable mechanisms from task-specific tuning (Can harness modules improve separately from benchmark data?). The summaries of the large-gain results above don't say whether they passed either check, so treat the size of those gains as an upper bound.
Harness gains don't always show up as higher scores. Four mechanisms found by optimizing across many environments (action execution, context compaction, observation handling, delegated reading) cut token traffic by 44.7–49.0% at comparable performance (Can agent harnesses be automatically optimized across many environments?). The corpus supports this much: harness design plausibly accounts for anywhere from a few points to a large share of a score. That share is largest for weaker or mid-tier models, and no note yet measures it cleanly.
Sources 10 notes
DarwinX achieves average 17-point gains across benchmarks by evolving harness variants (prompts, tools, skills, control flow) under a preserve-and-extend contract while keeping the model frozen. Key evidence includes Terminal-Bench 2.1 rising to 84.7% and WebArena-Infinity reaching 93.0% audit-clean pass@1.
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
A controlled study varying planning, action space, and context management across models and budgets found that context management becomes most valuable under tight windows, while planning shifts from helping weaker models to cutting costs for stronger ones.
Show all 10 sources
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
A harness strict enough for one model over-blocks another, while policies general enough to transfer across domains miss application-specific safety relations. Domain semantics and model characteristics jointly determine which harness is effective.
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
ModularRSI evolves harness modules independently using contrastive trajectories on benchmark-disjoint data, showing consistent gains across unseen tasks and domains. The approach isolates mechanism-level improvements from task-specific adaptation by aggregating evidence across tasks before updating components.
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
- ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement