Why do some AI setups let the model do its own thing, while others make it fight the tools?
What makes a harness low-friction for model strategy?
This explores what makes the scaffolding around a model (its prompts, tools, and control flow) easy for the model to work through instead of fight, so the model's own strategy comes through.
This explores what makes the scaffolding around a model (its prompts, tools, and control flow) easy for the model to work through instead of fight. The corpus never defines 'low-friction' directly. Read together, though, the notes point to three things: the harness takes on work the model would otherwise burn effort on, it fits the model that's using it, and its contents are real strategy rather than clutter.
The first is moving work out of the model's head and into the harness's structure. Organizing a repository around runtime behavior, with an explicit map from behavior to code, raised win rates by 10–19 points and cut planner tokens by 8–13%. A weaker planner using the map matched stronger models at finding the right code Can explicit behavior maps help weaker planners compete with stronger models?. In another case, a stronger model built an inference harness that nearly doubled a weaker model's Theory-of-Mind scores. It did this mostly by moving unstable reasoning into deterministic code and routing tasks, not by asking for more thinking Can a stronger model lift a weaker one at test time without retraining?. Automated search across many environments landed on four plumbing mechanisms: action execution, context compaction, observation handling, and delegated reading. Together they cut token traffic by roughly 45–49% at comparable performance Can agent harnesses be automatically optimized across many environments?. In each case, friction was effort spent on plumbing instead of strategy.
The second is fit, and it varies by model. Models of every tier are about equally good at writing useful harness edits. But the benefit from those edits follows an inverted U that peaks in mid-tier models. Weak models often fail to invoke the harness at all, and strong models struggle to follow its instructions faithfully Do stronger models always evolve harnesses better?. So friction can come from a model that's too limited to use the harness or too independent to defer to it. Safety harnesses show the same mismatch: one strict enough for one model over-blocks another, so they need tuning per deployment Should safety harnesses be customized for each deployment?. Execution-level harnesses look more portable. One optimized runbook lifted several models on Terminal-Bench 2.1 without touching weights and carried over to newer models unchanged Can execution harnesses lift model performance without retuning weights?. This suggests mechanics travel across models better than policies do.
The third is whether the harness holds strategy or just cached fixes. Analysis of evolved harnesses found the edits were sensible but mostly memorized task-specific patches an agent could rediscover in a single rollout. Those add weight without converting hard failures into successes Do harness edits learn reusable strategies or memorize task fixes?. Two design ideas push the other way. Evolving modules separately, on data disjoint from the benchmark, isolates reusable improvements from task-specific ones Can harness modules improve separately from benchmark data?. A preserve-and-extend contract keeps proven behavior from being overwritten. Under that contract, evolving harness variants with a frozen model lifted results by about 17 points on average Can frozen models improve by evolving their harnesses?.
To tell whether a harness is helping or just spending more compute, compare it against test-time search with the same feedback and inference budget. Only gains beyond that baseline count as design value How should we measure gains from automatic harness evolution?. The friction question may also run in both directions. A deployed routing harness records trajectories and difficulty estimates that can become training data, so the model can learn to fit the harness as well Can a routing harness generate its own training data automatically?. One framing treats the versioned loop of model, harness, and contract as the unit that adapts, not any single model snapshot Where does model adaptation actually happen?.
Sources 12 notes
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
A harness strict enough for one model over-blocks another, while policies general enough to transfer across domains miss application-specific safety relations. Domain semantics and model characteristics jointly determine which harness is effective.
Show all 12 sources
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
Analysis of evolved harness trajectories shows rational, well-motivated edits across prompt and tool layers, but most persist fixes an agent could rediscover in a single rollout. Gains remain limited because memorized shortcuts cache what's already within reach rather than converting hard failures into successes.
ModularRSI evolves harness modules independently using contrastive trajectories on benchmark-disjoint data, showing consistent gains across unseen tasks and domains. The approach isolates mechanism-level improvements from task-specific adaptation by aggregating evidence across tasks before updating components.
DarwinX achieves average 17-point gains across benchmarks by evolving harness variants (prompts, tools, skills, control flow) under a preserve-and-extend contract while keeping the model frozen. Key evidence includes Terminal-Bench 2.1 rising to 84.7% and WebArena-Infinity reaching 93.0% audit-clean pass@1.
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
A deployed routing system records execution trajectories, capability demand estimates, and outcome data that can be converted into labeled training examples for fine-tuning and distillation, turning the harness into both a serving component and a difficulty labeler.
Macaron-V1 argues adaptation is a property of the recursive cycle linking model, harness, and external contract—not individual model snapshots. Weight updates are gated by audit and evaluation against an external contract, making the loop the unit of improvement and release.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Rethinking the Evaluation of Harness Evolution for Agents
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable