Can you make an AI's toolkit and prompts better on their own, or do they only work tuned to one specific model?
Can a harness design be optimized independently of the model inside it?
This explores whether you can improve the scaffolding around an AI model (its prompts, tools, memory handling and control flow) as a separate project, or whether a good harness only works with the particular model running inside it.
This explores whether the scaffolding around a model (its prompts, tools, context handling and control logic) can be improved as a separate piece of engineering, or whether harness and model have to be tuned together. The corpus gives a split answer, and where it splits is the interesting part. Some harness improvements carry over between models and some don't, and they tend to fall along a line between efficiency and capability.
The case for independence is strong. Evolving harness variants while keeping the model completely frozen produced average gains of about 17 points across benchmarks Can frozen models improve by evolving their harnesses?. When frontier models were allowed to choose between fixing their harness and training themselves harder, harness fixes kept helping while extra fine-tuning lowered scores in most comparisons Do harness fixes or heavier training drive frontier model gains?. Harness modules evolved on data kept separate from the benchmarks still improved unseen tasks, which suggests some harness ideas are general mechanisms and not tricks fitted to one test Can harness modules improve separately from benchmark data?. That's part of why Weng argues the near-term route to AI improving itself runs through harness engineering, not models rewriting their own weights Does recursive self-improvement start with harness engineering?.
The case against is just as concrete. Swapping between three production coding harnesses across five models barely changed accuracy on SWE-bench Verified, staying within normal rerun noise, but it changed token cost by up to three times Does switching between coding harnesses change model accuracy?. A large cross-environment optimization effort found something similar: the mechanisms it discovered cut token traffic nearly in half while performance stayed about the same Can agent harnesses be automatically optimized across many environments?. Put together, these suggest that the harness gains that transfer cleanly are mostly about cost and efficiency. Accuracy gains depend more on the model. Controlled experiments show this directly. Planning helps weaker models get answers right but mostly saves money for stronger ones, and context management matters most when the context window is tight Which coding harness components matter most in different conditions?. Safety harnesses are the extreme case: a policy strict enough for one model over-blocks another Should safety harnesses be customized for each deployment?.
The most surprising finding is about which models benefit. Models at every tier write roughly equally useful harness edits, but how much a model gains from a better harness follows an inverted U. Weak models fail to use the scaffolding, and strong models are less faithful at following it, so mid-tier models gain the most Do stronger models always evolve harnesses better?. That makes "harness for model X, built by model Y" a real design pattern. A stronger model building deterministic code and task routing for a weaker one nearly doubled the weaker model's Theory-of-Mind scores Can a stronger model lift a weaker one at test time without retraining?. One line of work treats the harness as temporary: first extend the scaffold, then train what it reveals into the model's weights, then remove the scaffold Can sequencing edits across three surfaces avoid hidden costs?.
One caution before trusting any claim of independent gains: many "harness improvements" may just be extra computation. Only gains beyond what equal-budget test-time search would achieve can be credited to the design itself How should we measure gains from automatic harness evolution?. So the short answer is that you can optimize a harness separately from its model, but mainly for efficiency. For accuracy and safety, the harness and the model have to be tuned together.
Sources 12 notes
DarwinX achieves average 17-point gains across benchmarks by evolving harness variants (prompts, tools, skills, control flow) under a preserve-and-extend contract while keeping the model frozen. Key evidence includes Terminal-Bench 2.1 rising to 84.7% and WebArena-Infinity reaching 93.0% audit-clean pass@1.
Across six frontier models in RSIGym's Joint track, harness fixes consistently improved performance while additional fine-tuning lowered scores in eight of ten comparisons. Claude agents ranked highest by spending more time diagnosing and revising harnesses rather than scaling training.
ModularRSI evolves harness modules independently using contrastive trajectories on benchmark-disjoint data, showing consistent gains across unseen tasks and domains. The approach isolates mechanism-level improvements from task-specific adaptation by aggregating evidence across tasks before updating components.
Weng argues RSI's initial path moves through optimizing deployment harnesses—instruction prompts to harness code to optimizer code—rather than models directly rewriting weights. This staged progression mirrors how prompt engineering gave way to instruction tuning while interface needs persisted.
Across three production harnesses and five models on SWE-bench Verified, switching harnesses shifted pass rates within normal rerun variation but changed token cost up to threefold. The cost difference came from fixed overhead in system prompts and tool schemas, multiplied by step count.
Show all 12 sources
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
A controlled study varying planning, action space, and context management across models and budgets found that context management becomes most valuable under tight windows, while planning shifts from helping weaker models to cutting costs for stronger ones.
A harness strict enough for one model over-blocks another, while policies general enough to transfer across domains miss application-specific safety relations. Domain semantics and model characteristics jointly determine which harness is effective.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
MetaRSI-v1 shows that a four-step composition—extend scaffold, amplify what it reveals, internalize into weights, retire extension—retains capability while avoiding the per-surface costs that single-edit operators impose. Each surface pays a different hidden price; the sequence distributes and recovers them.
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Sharpening Tax in Post-Training
- What Does a Harness Buy? Tokens, Mostly
- Harness Engineering for Self-Improvement
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution