INQUIRING LINE

The AI models that benefit most from saved shortcuts aren't the strongest ones — they're the ones in the middle.

Why do mid-tier models benefit more from memorized harness shortcuts?

This explores why the ability to *gain* from cached harness fixes (memorized shortcuts an agent writes into its own scaffolding) peaks in the middle of the capability range rather than rising with model strength.


This explores why memorized harness shortcuts — the task-specific fixes an agent writes into its own scaffolding — help mid-tier models most, rather than helping the strongest models most. The cleanest answer in the corpus is that two different abilities pull in opposite directions, and mid-tier models sit at the sweet spot where both are satisfied. The capacity to *produce* a useful harness edit turns out to be roughly flat across model tiers, but the capacity to *benefit* from those edits follows an inverted-U: weak models never reliably invoke the harness in the first place, while the strongest models struggle to follow the cached instruction faithfully once it's there Do stronger models always evolve harnesses better?. Mid-tier models are competent enough to trigger the shortcut and obedient enough to just run it.

That picture gets sharper once you look at what these edits actually are. Inspection of evolved harnesses shows most edits don't distill a transferable strategy — they cache a fix for a task the agent could already solve, persisting a shortcut rather than converting a failure into a success evolved-harness-edits-mostly-memorize-task-specific-fixes-rather-than-distillin. A memorized shortcut is a crutch, and crutches help exactly the class of model that can walk but not sprint. The strongest models have less to gain because they can rediscover the fix in a single rollout; the weakest can't use the crutch at all.

There's a parallel result on the planning side that reinforces this: giving a weaker planner an explicit behavior-to-code map let it match stronger models on localization while spending fewer tokens Can explicit behavior maps help weaker planners compete with stronger models?. Externalized, memorized structure substitutes for raw capability — it lifts the middle of the distribution toward the top, but the top was already there on its own. The same substitution logic explains why the benefit curve bends downward at the high end.

A cautionary thread runs underneath all of this. Because harness evolution is itself a search loop, apparent gains are confounded with sheer search effort and can amount to memorizing the test set unless you hold the search budget fixed and evaluate on held-out tasks Are harness evolution gains really from better design?. Some of the mid-tier 'benefit' may be shortcut-caching that doesn't generalize — which rhymes with a broader failure mode where models reinforced on the wrong signal learn degenerate shortcuts (answer repetition, computation-skipping) that actively contaminate real capability Do overly hard RLVR samples actually harm model capabilities?.

The thing you might not have expected to learn: the reason the strongest models benefit *less* isn't that they're too good for shortcuts — it's an instruction-following problem. A memorized harness edit is a rigid output-format prescription, and there's evidence that what such prescriptions transfer is knowledge of the output space, not task understanding Does instruction tuning teach task understanding or output format?. Strong models, with richer priors of their own, are more prone to deviate from the cached format than to slavishly execute it — so the very sophistication that lets them solve tasks unaided is what makes them worse at cashing in someone else's memorized answer.


Sources 6 notes

Do stronger models always evolve harnesses better?

Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.

Can explicit behavior maps help weaker planners compete with stronger models?

A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.

Are harness evolution gains really from better design?

Harness evolution is itself a search loop, so its reported improvements are confounded with search effort. Only improvements beyond equal task-level search budget can be attributed to harness design, and held-out evaluation is needed to rule out task memorization.

Do overly hard RLVR samples actually harm model capabilities?

Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.

Does instruction tuning teach task understanding or output format?

Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.

Research prompt for your LLMexpand ↓

Copy into ChatGPT or Claude to take this line of inquiry further — it asks the model to find newer work and re-test which earlier constraints still hold.

You are a research analyst investigating a still-open question: why do mid-tier models benefit more from memorized harness shortcuts than the strongest models do?

What a curated library found — and when (dated claims, not current truth; spanning ~2023–2026):
- The capacity to *produce* a useful harness edit is roughly flat across tiers, but the capacity to *benefit* follows an inverted-U: weak models never reliably invoke the harness, strong models deviate from the cached instruction (~2026).
- Most evolved edits don't distill transferable strategy — they cache a task-specific fix the agent could already solve, persisting a shortcut not a lesson (~2026).
- Giving a weaker planner an explicit behavior-to-code map matched stronger models on localization while spending fewer tokens (~2026).
- Apparent gains confound with sheer search effort — can amount to memorizing the test set unless search budget is fixed and held-out tasks used (~2026).
- Rigid format prescriptions transfer knowledge of output space, not task understanding — so strong models' richer priors make them worse at cashing in a cached answer (~2023).

Anchor papers (verify; mind their dates): Harness Updating Is Not Harness Benefit (arXiv:2605.30621, 2026); Code as Agent Harness (arXiv:2605.18747, 2026); Scaling the Harness in Agentic AI (arXiv:2605.26112, 2026); Do Models Really Learn to Follow Instructions? (arXiv:2305.11383, 2023).

Your task: (1) RE-TEST EACH CONSTRAINT — for every finding, judge whether newer models, training, tooling, orchestration (memory, caching, multi-agent), or evaluation have RELAXED or OVERTURNED it; separate the durable question from the perishable limit, cite what resolved it, and say plainly where a constraint still holds. (2) Surface the strongest CONTRADICTING or SUPERSEDING work from the last ~6 months, foregrounding disagreement over whether the inverted-U survives stronger frontier models. (3) Propose 2 research questions that assume the regime has moved.

Cite arXiv IDs; flag anything you cannot ground in a real paper.