Should an AI agent learn its playbook by training a model, or by having another AI rewrite the notes?
Can RL-trained policies outperform text-space optimizers for evolving skill repositories?
This explores whether training a model with reinforcement learning to curate a library of agent skills beats having a text-based optimizer rewrite the skill documents directly.
This explores whether training a model with reinforcement learning to curate a library of agent skills beats having a text-based optimizer rewrite the skill documents directly. The corpus has no head-to-head comparison. It does have one paper on each side, and they win on different things.
The text-space camp treats the skill document as the thing being trained. SkillOpt keeps the agent frozen and proposes edits to the document in plain text. It accepts an edit only if it improves performance on held-out validation tasks. Across 52 benchmark cells and seven models it matches or beats baselines, adds zero inference cost, and transfers across models (Can skill documents be optimized like neural network weights?). This descends from older ideas in the collection. Reflexion showed that an agent can improve from written self-diagnoses alone, and that clear success/failure feedback stops it from rationalizing (Can agents learn from failure without updating their weights?). VOYAGER showed that a growing library of executable skills avoids the forgetting that weight updates cause (Can agents learn new skills without forgetting old ones?). Mind Evolution showed that text-only search, with LLM-written mutations and crossover, can beat simple sampling and revision (Can evolutionary search beat sampling and revision at inference time?).
The RL camp changes what gets learned. SkillOS trains a separate curator with RL and leaves the executor frozen. Trained across streams of tasks, the curator moves the repository away from generic, verbose additions. It moves toward actionable execution logic and cross-task meta-strategies, and it still works when you swap in a different executor model or domain (Can a separate trained curator improve skill libraries better than frozen agents?). The gain is a learned editing policy. The curator has learned what kind of skill is worth keeping, so it does not have to search for that from scratch on each new repository.
There is a plausible catch, which is an inference from other notes and not something SkillOS tests. RL has a habit of narrowing whatever it trains. Low reward variance pushes policies toward generic, input-agnostic templates (Why do language models collapse into generic templates?). RL also squeezes exploration diversity in search agents (Does reinforcement learning squeeze exploration diversity in search agents?). And RL post-training collapses onto one dominant format (Does RL training collapse format diversity in pretrained models?). A curator that drifts toward one house style of skill would be a quiet failure. SkillOpt's validation gate can't drift this way, because it never touches weights and rejects any edit that doesn't help.
So the corpus can't say that RL policies outperform text-space optimizers. It says they trade differently. SkillOpt costs nothing to train and is checked at every edit, but it optimizes one skill document at a time. SkillOS pays for training once and gets a curator that generalizes across executors. The comparison the corpus is missing is both methods run on the same benchmarks, with the RL training cost counted.
Sources 8 notes
SkillOpt treats skill documents as trainable external state of frozen agents, using a text-space optimizer with held-out validation gating to accept only edits that improve performance. Across 52 benchmark cells and seven models, the approach matches or exceeds baselines while adding zero inference cost and enabling transfer across models.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
Mind Evolution, an evolutionary search strategy using LLM-generated crossover and mutation with island model diversity, solves 98%+ of planning tasks and significantly outperforms best-of-N and sequential revision strategies while working directly in natural language without task formalization.
SkillOS shows that separating a trainable curator from a frozen executor, grouped by task streams, causes skill repositories to shift from generic verbose additions toward actionable execution logic and cross-task meta-strategies. The trained curator generalizes across different executor backbones and domains.
Show all 8 sources
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- SkillClaw: Let Skills Evolve Collectively with Agentic Evolver
- Demystifying Agent Skills: Why They Work-Until They Don't
- MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation
- SkillOS: Learning Skill Curation for Self-Evolving Agents
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- RAGEN-2: Reasoning Collapse in Agentic RL
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?