Fixing an AI agent's behavior isn't hard — the bottleneck is finding all the scattered places that behavior actually lives.
What makes behavior localization the bottleneck in agent harness evolution?
This explores why the hard part of improving an agent's scaffolding (its prompts, tools, and orchestration code) isn't writing better edits but finding all the scattered places where a single behavior actually lives.
This explores why the hard part of improving an agent's scaffolding isn't writing better edits but finding all the scattered places where a single behavior actually lives. The core claim is a representational mismatch: when you ask a harness to change one behavior, that behavior isn't sitting in one function — it's smeared across files, tool definitions, prompt fragments, and pipeline stages. So the agent evolving the harness spends most of its effort just locating every site that implements the behavior, and any edit that misses a location leaves the old behavior partly intact Why is finding distributed behavior code so hard?. Generation was never the constraint; localization is.
The cleanest evidence that localization is the true bottleneck comes from what happens when you remove it. Reorganizing a harness around runtime behavior — building an explicit behavior-to-code map — let *weaker* planners match the code-localization precision and recall of much stronger models, while lifting win rates 10–19 points and cutting planner tokens Can explicit behavior maps help weaker planners compete with stronger models?. That's the tell: if handing a model a map closes most of the gap, then what the stronger model was buying you all along was better search through distributed code, not better ideas about what to change.
This also reframes a puzzle about model tiers. The raw capacity to *propose* useful harness edits is roughly flat across model sizes — but the capacity to actually *benefit* peaks in the middle, an inverted U Do stronger models always evolve harnesses better?. Localization difficulty explains both tails: weak models can't reliably find and invoke the right code paths, and even strong models get tripped up threading a change faithfully through every distributed site. The bottleneck isn't intelligence about the edit; it's the structural work surrounding it.
Worse, when localization is hard, the path of least resistance is to stop generalizing. Inspection of evolved harnesses shows most edits just cache task-specific fixes the agent could have rediscovered in a single rollout, rather than distilling a reusable strategy Do harness edits learn strategies or just memorize fixes?. If you can't cleanly find where a behavior lives, you patch the one spot that failed on this task instead of restructuring the behavior everywhere — memorization is what localization failure looks like from the outside. And because harness evolution is itself a search loop, apparent gains have to be measured against a matched search budget, or you're crediting design for what was really just more attempts at finding the right spot Are harness evolution gains really from better design?.
Step back and the bottleneck marks a fault line in self-improving agents generally. These systems split into a slow parametric loop (updating weights) and a fast non-parametric loop (updating prompts, memory, tools), and progress has rushed toward the fast loop because scaffold edits are cheap and reversible Do self-improving agents really split into two distinct loops?. But the fast loop only pays off if edits land coherently across the whole scaffold — which is exactly where distributed behavior defeats it. It's worth noticing that the alternative traditions dodge the problem by construction: skill libraries like VOYAGER externalize each capability as one indexed, composable unit so a behavior has a single home Can agents learn new skills without forgetting old ones?, and memory-based agents route improvement through discrete memory operations rather than surgery on tangled code Can agents learn continuously from experience without updating weights?. The lesson underneath: harnesses are hard to evolve not because they're complex but because they scatter one idea across many places, and the fix is representational — give each behavior an address before you try to change it.
Sources 8 notes
The core difficulty in evolving production harnesses is not generating edits but finding every code location that implements a behavior. Harnesses distribute single behaviors across files, functions, and stages, creating a representational mismatch between behavioral requests and structural code organization.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Inspecting evolved harness edits reveals rational changes across prompt and tool layers, but most persist fixes the agent could rediscover in one rollout. Gains remain limited because edits cache shortcuts for already-solvable tasks rather than converting failures into successes.
Harness evolution is itself a search loop, so its reported improvements are confounded with search effort. Only improvements beyond equal task-level search budget can be attributed to harness design, and held-out evaluation is needed to rule out task memorization.
Show all 8 sources
A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- Rethinking the Evaluation of Harness Evolution for Agents
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
- Code as Agent Harness
- SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
- Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments