The AI models that benefit most from saved shortcuts aren't the strongest ones — they're the ones in the middle.
Why do mid-tier models benefit more from memorized harness shortcuts?
This explores why the ability to *gain* from cached harness fixes (memorized shortcuts an agent writes into its own scaffolding) peaks in the middle of the capability range rather than rising with model strength.
This explores why memorized harness shortcuts — the task-specific fixes an agent writes into its own scaffolding — help mid-tier models most, rather than helping the strongest models most. The cleanest answer in the corpus is that two different abilities pull in opposite directions, and mid-tier models sit at the sweet spot where both are satisfied. The capacity to *produce* a useful harness edit turns out to be roughly flat across model tiers, but the capacity to *benefit* from those edits follows an inverted-U: weak models never reliably invoke the harness in the first place, while the strongest models struggle to follow the cached instruction faithfully once it's there Do stronger models always evolve harnesses better?. Mid-tier models are competent enough to trigger the shortcut and obedient enough to just run it.
That picture gets sharper once you look at what these edits actually are. Inspection of evolved harnesses shows most edits don't distill a transferable strategy — they cache a fix for a task the agent could already solve, persisting a shortcut rather than converting a failure into a success evolved-harness-edits-mostly-memorize-task-specific-fixes-rather-than-distillin. A memorized shortcut is a crutch, and crutches help exactly the class of model that can walk but not sprint. The strongest models have less to gain because they can rediscover the fix in a single rollout; the weakest can't use the crutch at all.
There's a parallel result on the planning side that reinforces this: giving a weaker planner an explicit behavior-to-code map let it match stronger models on localization while spending fewer tokens Can explicit behavior maps help weaker planners compete with stronger models?. Externalized, memorized structure substitutes for raw capability — it lifts the middle of the distribution toward the top, but the top was already there on its own. The same substitution logic explains why the benefit curve bends downward at the high end.
A cautionary thread runs underneath all of this. Because harness evolution is itself a search loop, apparent gains are confounded with sheer search effort and can amount to memorizing the test set unless you hold the search budget fixed and evaluate on held-out tasks Are harness evolution gains really from better design?. Some of the mid-tier 'benefit' may be shortcut-caching that doesn't generalize — which rhymes with a broader failure mode where models reinforced on the wrong signal learn degenerate shortcuts (answer repetition, computation-skipping) that actively contaminate real capability Do overly hard RLVR samples actually harm model capabilities?.
The thing you might not have expected to learn: the reason the strongest models benefit *less* isn't that they're too good for shortcuts — it's an instruction-following problem. A memorized harness edit is a rigid output-format prescription, and there's evidence that what such prescriptions transfer is knowledge of the output space, not task understanding Does instruction tuning teach task understanding or output format?. Strong models, with richer priors of their own, are more prone to deviate from the cached format than to slavishly execute it — so the very sophistication that lets them solve tasks unaided is what makes them worse at cashing in someone else's memorized answer.
Sources 6 notes
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
Harness evolution is itself a search loop, so its reported improvements are confounded with search effort. Only improvements beyond equal task-level search budget can be attributed to harness design, and held-out evaluation is needed to rule out task memorization.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- Rethinking the Evaluation of Harness Evolution for Agents
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
- Code as Agent Harness
- Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning
- A Survey on Post-training of Large Language Models