AI agents keep rewriting their own instructions and tools, but mostly to memorize one-off fixes. Can that be steered toward lessons that transfer?
Can harness evolution be redirected from memorization toward strategy distillation?
This explores whether the automated rewriting of an agent's scaffolding (its prompts, tools and helper code) can be steered so that the edits capture transferable strategies instead of one-off patches for individual tasks.
This explores whether the automated rewriting of an agent's scaffolding (its prompts, tools and helper code) can be steered so that the edits capture transferable strategies instead of one-off patches for individual tasks. The corpus documents the problem clearly but has no note showing a working redirect. Evolved harness edits are usually rational and well motivated, yet most of them save a fix the agent could have rediscovered in a single rollout, so they cache what was already within reach instead of turning hard failures into successes Do harness edits learn reusable strategies or memorize task fixes?. Neighboring work points to three places to intervene.
The first is what gets written down. SkillRL treats successes and failures differently: wins are kept as concrete demonstrations, and failures are boiled down into abstracted lessons. That approach beat uniform consolidation while using less context Should successful and failed episodes be processed differently?. Memorizing harnesses do the reverse and turn each failure into its own specific patch. The ACE framework shows how to keep a growing context from degrading. It uses small generation, reflection and curation updates instead of full rewrites, so detail isn't lost to compression Can context playbooks prevent knowledge loss during iteration?. Voyager adds a test for what counts as a strategy. Its skills are stored so that later, more complex skills can be built from them, and something that gets reused and composed is unlikely to be a one-task patch Can agents learn new skills without forgetting old ones?.
The second is what the search is made to explore. RLAD trains a model to propose abstractions (short strategy hints) before solving. Spending test-time compute on diverse abstractions beat sampling more solutions once budgets got large, because abstractions force breadth where depth alone stalls Can abstractions guide exploration better than depth alone?. Separating a decomposer from a solver gives a related clue: decomposition skill transferred across domains, while solving skill did not Does separating planning from execution improve reasoning accuracy?. That suggests strategy-level content generalizes and task-level fixes don't. How the harness is organized also matters. Mapping behavior to code let weaker planners match stronger models at finding the right code, with win rates up 10–19 points Can explicit behavior maps help weaker planners compete with stronger models?. Diversity has to be protected on purpose, though. Evolutionary search with island-style populations does that Can evolutionary search beat sampling and revision at inference time?. Reward-only optimization tends to squeeze exploration into a few narrow winning moves Does reinforcement learning squeeze exploration diversity in search agents?, and a harness loop that scores only pass rate may do the same.
The third is who does the evolving. The ability to write useful harness edits is flat across model sizes, but the ability to benefit from them peaks in mid-tier models. Weak models fail to invoke the harness, and strong ones follow its instructions unfaithfully Do stronger models always evolve harnesses better?. So a strategy-level edit is only useful if the model running it can follow it. Separately, RL training tends to fix execution correctness first, and only then does strategic planning become the bottleneck. Concentrating optimization on planning tokens is what pays off in the second phase Does RL training follow a predictable two-phase learning sequence?. This is my reading rather than something the notes test directly. Memorized fixes look like the first kind of gain, and redirecting harness evolution would mean aiming its edits at planning-level content.
The corpus doesn't yet show that these levers work on harness evolution itself. What it does offer is a diagnostic. If an edit is something the agent could rediscover in one rollout, it's memorization. If it makes a hard failure succeed, or gets reused across tasks, it's closer to a strategy.
Sources 11 notes
Analysis of evolved harness trajectories shows rational, well-motivated edits across prompt and tool layers, but most persist fixes an agent could rediscover in a single rollout. Gains remain limited because memorized shortcuts cache what's already within reach rather than converting hard failures into successes.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
The ACE framework treats contexts as evolving playbooks using generation-reflection-curation loops rather than full rewrites. This prevents knowledge loss from compression and detail erosion, achieving +10.6% on agentic tasks and +8.6% on finance without labeled supervision.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
RLAD jointly trains abstraction and solution generators, showing that allocating test-time compute to diverse abstractions outperforms parallel solution sampling at large budgets. Abstractions create structured breadth-first exploration that prevents the underthinking failure mode of depth-only reasoning chains.
Show all 11 sources
Modular architectures with separate decomposer and solver models outperform monolithic LLMs, with decomposition ability transferring across domains while solving ability does not. The separation prevents planning-execution interference and produces more generalizable skills.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
Mind Evolution, an evolutionary search strategy using LLM-generated crossover and mutation with island model diversity, solves 98%+ of planning tasks and significantly outperforms best-of-N and sequential revision strategies while working directly in natural language without task formalization.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- Rethinking the Evaluation of Harness Evolution for Agents
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
- Reasoning LLMs are Wandering Solution Explorers
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution