Why do skills an AI agent picks up through training stay locked to the exact scenario it trained on?
What stops evolved agent behaviors from generalizing beyond specific tasks?
This explores why behaviors an agent picks up through training or optimization often stay glued to the exact task they were learned on — and what the corpus says the culprits are.
This explores why behaviors an agent picks up through training or optimization often stay glued to the exact task they were learned on. The corpus points to a few distinct bottlenecks, and the interesting part is that they're not the same bottleneck wearing different hats — they're different failure modes, and the fix for one won't touch the others.
The first is a data ceiling. When agents learn from static expert demonstrations, their competence is capped by whatever scenarios the dataset's curator happened to imagine — they never touch the environment during training, so they can't learn from their own failures or stretch past what was shown Can agents learn beyond what their training data shows?. Generalization can't exceed a ceiling nobody built above the training set.
The second is subtler and almost the opposite problem: reinforcement learning actively narrows behavior. Instead of a ceiling, you get a funnel — RL pushes policies to converge on a few reward-maximizing strategies, collapsing the behavioral diversity that would let an agent adapt to a new task. This 'entropy collapse' shows up in search agents the same way it does in reasoning, and notably, supervised fine-tuning on diverse demonstrations preserves the exploration breadth that RL squeezes out Does reinforcement learning squeeze exploration diversity in search agents?. So the very process meant to make an agent good at a task can make it worse at everything adjacent.
What's striking is that the corpus's success stories route *around* weight updates entirely. VOYAGER stores executable skills in an indexed library and composes complex skills from simpler ones, learning continuously without the catastrophic forgetting that weight-based methods suffer Can agents learn new skills without forgetting old ones?. Agent Workflow Memory generalizes better precisely because it abstracts *sub-task* routines rather than whole tasks — stripping out example-specific values so the routine transfers, with gains that grow as the train-test gap widens Can agents learn reusable sub-task routines from past experience?. The lesson hiding here: generalization improves when you store behavior at a *finer* granularity than the task, not the task itself. Reflexion and AgentFly make the same bet from the memory side — improvement lives in episodic memory or a memory-augmented decision process, not in the frozen weights Can agents learn from failure without updating their weights? Can agents learn continuously from experience without updating weights?.
A survey framework ties this together: self-improving agents split into a slow parametric loop that rewrites model weights and a fast non-parametric loop that rewrites prompts, memory, and tools — and progress is concentrating in the fast loop because scaffold edits are cheap and reversible while weight updates are neither Do self-improving agents really split into two distinct loops?. But even the fast loop has a ceiling you might not expect: the ability to *produce* useful harness edits is flat across model sizes, yet the ability to *benefit* from them peaks at mid-tier models — weak models can't invoke the improvements, and the strongest ones struggle to follow their own instructions faithfully Do stronger models always evolve harnesses better?. So what stops evolved behavior from generalizing isn't one wall — it's a data ceiling, a diversity funnel, and the choice of *where* you store what was learned, with an added twist that being a stronger model doesn't automatically let you cash in your own upgrades.
Sources 8 notes
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
Agent Workflow Memory induces sub-task routines at finer granularity than full tasks, abstracts example-specific values, and compounds them hierarchically. This produces 24.6% relative gain on Mind2Web and 51.1% on WebArena, with larger gains as train-test gaps widen.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
Show all 8 sources
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- SkillClaw: Let Skills Evolve Collectively with Agentic Evolver
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- Useful Memories Become Faulty When Continuously Updated by LLMs
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
- Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
- Self-Improvements in Modern Agentic Systems: A Survey
- MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation