When an AI rewrites its own how-to notes, why do vague, wordy advice turn into concrete step-by-step instructions?
Why do generic skill descriptions evolve into execution-oriented ones?
This explores why, when an AI system rewrites its own library of agent skills (reusable how-to documents), the entries drift from broad, wordy descriptions toward concrete, step-by-step execution logic.
This explores why, when an AI system rewrites its own library of agent skills (reusable how-to documents), the entries drift from broad, wordy descriptions toward concrete execution logic. No note in the corpus answers the "why" head-on. Several point at the same mechanism, though: the rewriting is judged by whether the agent's actions improve, and only concrete procedure changes actions.
The direct observation comes from SkillOS. A trainable curator, kept separate from a frozen executor, learns to move a skill repository away from "generic verbose additions" and toward actionable execution logic and cross-task meta-strategies (Can a separate trained curator improve skill libraries better than frozen agents?). The curator is rewarded on how the executor performs, not on how good a description reads. SkillOpt applies the same pressure more bluntly. It treats the skill document like model weights and accepts an edit only if it improves held-out performance (Can skill documents be optimized like neural network weights?). Text that doesn't change what the agent does gets filtered out.
Execution-oriented content survives because of what skills actually do. An analysis of 8,135 trials found that skills mostly work as procedural anchors, which stabilize the agent's actions. That accounted for 65.7% of skill cases, against 4.5% for injecting missing knowledge (Do skills teach procedures or inject missing facts?). A generic description tells a capable model nothing it didn't already know, while a concrete procedure gives it something to follow. The same note also explains why the meta-strategies show up. Skills fail when they're invoked out of context or followed too rigidly, so mature skills need guidance on when to apply them as well as how. An adjacent result points the same way. Models tuned on deliberately wrong or empty instructions performed about as well as those trained on correct ones, because what transfers is the shape of the output, not the semantic gloss (Does instruction tuning teach task understanding or output format?). Descriptive prose carries less usable signal than it appears to.
The same pull toward specifics appears in related work. Agent Workflow Memory extracts routines at sub-task level and strips out example-specific values. The gains grow as the gap between training and test tasks widens, so procedure at the right granularity is what transfers (Can agents learn reusable sub-task routines from past experience?). The generic end is the default resting place when nothing pushes back. When reward variance is low, policies collapse into input-agnostic templates (Why do language models collapse into generic templates?). Generic skill text looks like the same kind of collapse, and execution-level detail is what a real performance signal pulls out of it.
Two notes complicate the picture. Code2Skill skips the evolution entirely by lifting skills from source code, so they start out executable and carry their evidence with them (Can source code replace experience as skill raw material?). And evolving a good harness is a measurable skill in its own right. Models struggle to keep useful intermediate updates, and results shift sharply with the executor, so downstream scores alone can't tell you whether an evolved skill is good (Can language models build and maintain their own agent harnesses?). SkillOS is the counterweight here: its trained curator generalizes across executor backbones, which suggests the drift toward execution logic isn't tied to one model.
Sources 8 notes
SkillOS shows that separating a trainable curator from a frozen executor, grouped by task streams, causes skill repositories to shift from generic verbose additions toward actionable execution logic and cross-task meta-strategies. The trained curator generalizes across different executor backbones and domains.
SkillOpt treats skill documents as trainable external state of frozen agents, using a text-space optimizer with held-out validation gating to accept only edits that improve performance. Across 52 benchmark cells and seven models, the approach matches or exceeds baselines while adding zero inference cost and enabling transfer across models.
Analysis of 8,135 trials shows procedural anchoring accounts for 65.7% of skill cases versus 4.5% for knowledge injection. Skills fail when retrieved incorrectly, invoked out of context, or followed too rigidly.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Agent Workflow Memory induces sub-task routines at finer granularity than full tasks, abstracts example-specific values, and compounds them hierarchically. This produces 24.6% relative gain on Mind2Web and 51.1% on WebArena, with larger gains as train-test gaps widen.
Show all 8 sources
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Code2Skill extracts over 1 million auditable skills from GitHub repositories through source-blind reconstruction verification, improving performance from 42.90 to 47.90 macro-average across eight benchmarks, outperforming trajectory and document-derived alternatives.
Research shows that LLMs vary sharply in building harnesses across domains, struggle to retain useful intermediate updates during evolution, and produce harnesses whose performance shifts dramatically with different executors—demonstrating that harness quality cannot be inferred from downstream task scores alone.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Demystifying Agent Skills: Why They Work-Until They Don't
- RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources
- MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation
- SkillOS: Learning Skill Curation for Self-Evolving Agents
- Grounded Skill Synthesis from Code at Scale for Agentic Intelligence
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- SkillOpt: Executive Strategy for Self-Evolving Agent Skills
- Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose