INQUIRING LINE

Fixing an AI agent's behavior isn't hard — the bottleneck is finding all the scattered places that behavior actually lives.

What makes behavior localization the bottleneck in agent harness evolution?

This explores why the hard part of improving an agent's scaffolding (its prompts, tools, and orchestration code) isn't writing better edits but finding all the scattered places where a single behavior actually lives.


This explores why the hard part of improving an agent's scaffolding isn't writing better edits but finding all the scattered places where a single behavior actually lives. The core claim is a representational mismatch: when you ask a harness to change one behavior, that behavior isn't sitting in one function — it's smeared across files, tool definitions, prompt fragments, and pipeline stages. So the agent evolving the harness spends most of its effort just locating every site that implements the behavior, and any edit that misses a location leaves the old behavior partly intact Why is finding distributed behavior code so hard?. Generation was never the constraint; localization is.

The cleanest evidence that localization is the true bottleneck comes from what happens when you remove it. Reorganizing a harness around runtime behavior — building an explicit behavior-to-code map — let *weaker* planners match the code-localization precision and recall of much stronger models, while lifting win rates 10–19 points and cutting planner tokens Can explicit behavior maps help weaker planners compete with stronger models?. That's the tell: if handing a model a map closes most of the gap, then what the stronger model was buying you all along was better search through distributed code, not better ideas about what to change.

This also reframes a puzzle about model tiers. The raw capacity to *propose* useful harness edits is roughly flat across model sizes — but the capacity to actually *benefit* peaks in the middle, an inverted U Do stronger models always evolve harnesses better?. Localization difficulty explains both tails: weak models can't reliably find and invoke the right code paths, and even strong models get tripped up threading a change faithfully through every distributed site. The bottleneck isn't intelligence about the edit; it's the structural work surrounding it.

Worse, when localization is hard, the path of least resistance is to stop generalizing. Inspection of evolved harnesses shows most edits just cache task-specific fixes the agent could have rediscovered in a single rollout, rather than distilling a reusable strategy Do harness edits learn strategies or just memorize fixes?. If you can't cleanly find where a behavior lives, you patch the one spot that failed on this task instead of restructuring the behavior everywhere — memorization is what localization failure looks like from the outside. And because harness evolution is itself a search loop, apparent gains have to be measured against a matched search budget, or you're crediting design for what was really just more attempts at finding the right spot Are harness evolution gains really from better design?.

Step back and the bottleneck marks a fault line in self-improving agents generally. These systems split into a slow parametric loop (updating weights) and a fast non-parametric loop (updating prompts, memory, tools), and progress has rushed toward the fast loop because scaffold edits are cheap and reversible Do self-improving agents really split into two distinct loops?. But the fast loop only pays off if edits land coherently across the whole scaffold — which is exactly where distributed behavior defeats it. It's worth noticing that the alternative traditions dodge the problem by construction: skill libraries like VOYAGER externalize each capability as one indexed, composable unit so a behavior has a single home Can agents learn new skills without forgetting old ones?, and memory-based agents route improvement through discrete memory operations rather than surgery on tangled code Can agents learn continuously from experience without updating weights?. The lesson underneath: harnesses are hard to evolve not because they're complex but because they scatter one idea across many places, and the fix is representational — give each behavior an address before you try to change it.


Sources 8 notes

Why is finding distributed behavior code so hard?

The core difficulty in evolving production harnesses is not generating edits but finding every code location that implements a behavior. Harnesses distribute single behaviors across files, functions, and stages, creating a representational mismatch between behavioral requests and structural code organization.

Can explicit behavior maps help weaker planners compete with stronger models?

A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.

Do stronger models always evolve harnesses better?

Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.

Do harness edits learn strategies or just memorize fixes?

Inspecting evolved harness edits reveals rational changes across prompt and tool layers, but most persist fixes the agent could rediscover in one rollout. Gains remain limited because edits cache shortcuts for already-solvable tasks rather than converting failures into successes.

Are harness evolution gains really from better design?

Harness evolution is itself a search loop, so its reported improvements are confounded with search effort. Only improvements beyond equal task-level search budget can be attributed to harness design, and held-out evaluation is needed to rule out task memorization.

Show all 8 sources
Do self-improving agents really split into two distinct loops?

A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.

Can agents learn new skills without forgetting old ones?

VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.

Can agents learn continuously from experience without updating weights?

AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.

Research prompt for your LLMexpand ↓

Copy into ChatGPT or Claude to take this line of inquiry further — it asks the model to find newer work and re-test which earlier constraints still hold.

You are an agent-systems analyst. Still-open question: what makes behavior LOCALIZATION — not edit generation — the bottleneck in evolving an agent's harness?

What a curated library found — and when (dated claims, not current truth; findings span ~2025–2026):
- A behavior isn't one function; it's smeared across files, tool defs, prompt fragments, and pipeline stages, so most effort goes to finding every site, and any missed site leaves the old behavior partly intact (~2026).
- Reorganizing a harness around runtime behavior — an explicit behavior-to-code map — let WEAKER planners match strong models' localization precision/recall, lifting win rates 10–19 points while cutting planner tokens (~2026).
- Capacity to PROPOSE useful edits is roughly flat across model tiers, but capacity to BENEFIT peaks in the middle — an inverted U (~2026).
- Most evolved edits just cache task-specific fixes rediscoverable in one rollout, rather than distilling reusable strategy; gains must be judged against a matched search-budget baseline (~2026).

Anchor papers (verify; mind their dates): Agent Learning via Early Experience (arXiv:2510.08558, 2025); Code as Agent Harness (arXiv:2605.18747, 2026); Harness Updating Is Not Harness Benefit (arXiv:2605.30621, 2026); From Model Scaling to System Scaling (arXiv:2605.26112, 2026).

Your task:
(1) RE-TEST EACH CONSTRAINT. For every finding, judge whether newer models, training, tooling (SDKs, harnesses), orchestration (memory, caching, multi-agent), or evaluation has RELAXED or OVERTURNED it. Separate the durable question (likely still open) from the perishable limitation; cite what resolved it, and say plainly where a constraint still holds — e.g. does better code-search tooling dissolve the inverted-U?
(2) Surface the strongest CONTRADICTING or SUPERSEDING work from the last ~6 months.
(3) Propose 2 research questions that ASSUME the regime may have moved.

Cite arXiv IDs; flag anything you cannot ground in a real paper.