Swap the AI model running the same setup of prompts and tools, and results can change dramatically. So who gets the credit?
How much does executor choice change a harness's actual performance?
This explores how much the model that actually runs a harness (the executor that follows its prompts, tools, and control flow) changes the harness's results, and whether a harness's quality belongs to the harness or to the executor it was tested with.
This reads 'executor' as the model that runs a harness's steps, and asks how much swapping it changes the outcome. The corpus says it changes a lot, but it gives no single number. The most direct evidence is a study of whether models can build and maintain their own harnesses. It found that the same harness's performance shifts dramatically with different executors, so a harness's quality can't be read off one model's downstream score (Can language models build and maintain their own agent harnesses?). A harness is not a portable object with a fixed value.
Several other notes show why. A safety harness strict enough for one model over-blocks another, so which harness works depends on both the domain and the model (Should safety harnesses be customized for each deployment?). In coding harnesses, the same component plays different roles depending on who runs it. Planning helps weaker models, but for stronger models it mostly cuts cost. Context management pays off most when the window is tight (Which coding harness components matter most in different conditions?). Swapping the executor changes what each part of the harness is for, on top of changing the score.
The effect also doesn't track model strength in a straight line. Producing useful harness edits is about equally easy for every model tier, but benefiting from those edits follows an inverted U. Weak models often fail to invoke the harness at all, and the strongest models struggle to follow its instructions faithfully. So mid-tier models gain the most (Do stronger models always evolve harnesses better?). A stronger executor won't necessarily get a bigger boost from a better harness. Transfer can still work well. One execution-harness runbook carried over to newer models without modification, reaching 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points (Can execution harnesses lift model performance without retuning weights?). That is the same recipe with a different size of payoff for each model. Some harness gains may be less model-dependent. Four mechanisms found by scaled auto-research cut token traffic by 44.7–49.0% while keeping performance comparable, which suggests that efficiency gains sit apart from model quality (Can agent harnesses be automatically optimized across many environments?).
Two cautions apply when reading any harness number. First, a harness's gain only counts if it beats plain test-time search under the same budget, and otherwise you may just be measuring extra computation (How should we measure gains from automatic harness evolution?). Second, evolved harnesses often memorize task-specific fixes instead of learning general strategies, and they can overfit the tasks they were evolved on (Do harness edits learn reusable strategies or memorize task fixes?, Does harness self-improvement memorize tasks instead of learning broadly?). The notes don't test this directly, but a harness tuned on one executor plausibly memorizes that executor's habits too. That would help explain why moving it to another model shifts results so much.
The contrast with RL is worth noting. For reasoning, the choice of RL algorithm barely matters, because the pretrained model bounds what any optimizer can find (Does the choice of RL algorithm actually matter for reasoning?). Harnesses are the opposite. Population-level harness evolution on a frozen model can add about 17 points on average (Can frozen models improve by evolving their harnesses?). The corpus suggests that swapping the executor moves results by a comparable amount, so a harness should be evaluated across several executors before you trust its score.
Sources 11 notes
Research shows that LLMs vary sharply in building harnesses across domains, struggle to retain useful intermediate updates during evolution, and produce harnesses whose performance shifts dramatically with different executors—demonstrating that harness quality cannot be inferred from downstream task scores alone.
A harness strict enough for one model over-blocks another, while policies general enough to transfer across domains miss application-specific safety relations. Domain semantics and model characteristics jointly determine which harness is effective.
A controlled study varying planning, action space, and context management across models and budgets found that context management becomes most valuable under tight windows, while planning shifts from helping weaker models to cutting costs for stronger ones.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
Show all 11 sources
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
Analysis of evolved harness trajectories shows rational, well-motivated edits across prompt and tool layers, but most persist fixes an agent could rediscover in a single rollout. Gains remain limited because memorized shortcuts cache what's already within reach rather than converting hard failures into successes.
Recursive edits to agent scaffolding can memorize evolve-set tasks, causing in-distribution gains to shrink out-of-distribution. RRSI addresses this by constraining the proposal and selection stages to favor reusable mechanisms over benchmark-specific ones.
Expert Iteration, PPO, and RC-RL perform comparably on reasoning because exploration is constrained by the pretrained distribution, not the optimizer. RL functions as selection, not discovery—the prior contains most solutions the algorithm will find.
DarwinX achieves average 17-point gains across benchmarks by evolving harness variants (prompts, tools, skills, control flow) under a preserve-and-extend contract while keeping the model frozen. Key evidence includes Terminal-Bench 2.1 rising to 84.7% and WebArena-Infinity reaching 93.0% audit-clean pass@1.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
- Scaling Laws for Agent Harnesses via Effective Feedback Compute