Can an AI find the right how-to in a giant library built from videos, code and articles, then combine them?
How do agents retrieve and compose skills from hierarchical multimodal wikis?
This explores how an agent finds the right skill in a wiki-style library, built from mixed media like videos, code and articles and organized in levels, and how it combines skills into bigger jobs.
This explores how an agent finds the right skill in a wiki-style library, built from mixed media and organized in levels, and how it combines skills into bigger jobs. The collection has no single paper that shows the whole loop. It has the pieces, though, and they fit together better than you'd expect.
Start with what goes into the wiki. Resource2Skill distills tutorial videos, code repositories and articles into a Skill Wiki. Agents with access produced work 11.9 percentage points better across seven authoring domains Can tutorial videos teach software agents reusable skills?. Code2Skill shows the raw material doesn't even need to be human-made tutorials. It pulled over a million skills straight from GitHub code and checked each by rebuilding the code from the skill without seeing the source. Those skills beat ones derived from agent trajectories or documents Can source code replace experience as skill raw material?. Being multimodal here mostly means the wiki can be fed from many kinds of human know-how.
Hierarchy is what makes retrieval work, and the best evidence comes from books rather than skills. MegaRAG builds a multimodal knowledge graph with levels from high-level summaries down to page-specific details, and images are nodes in their own right. It answers cross-chapter questions that flat chunk retrieval can't reach Can multimodal knowledge graphs answer questions that flat retrieval cannot?. The natural transfer to a skill wiki is to route through the overview first and then pull the specific procedure. No note here tests that directly. A related idea is that matching should go by meaning rather than manual wiring. Versioned capability vectors in a vector index let agents discover each other's abilities this way, with policy and budget constraints applied at match time Can semantic capability vectors replace manual agent routing?.
Composition is where VOYAGER is the reference point. It stores executable skills in an embedding-indexed library and writes complex skills out of simpler ones already there. Environmental feedback refines them, and old skills survive because nothing in the model's weights is overwritten Can agents learn new skills without forgetting old ones?. MetaGPT adds a lesson about format. Agents coordinate better through standardized artifacts that they actively pull from a shared environment than through conversation Does structured artifact sharing outperform conversational coordination?. A consistent skill format is what lets pieces snap together.
The risk is in retrieval. An analysis of 8,135 trials found skills mostly work as procedural anchors, at 65.7% of cases, rather than by injecting missing facts, at 4.5%. They fail when retrieved wrongly, invoked out of context, or followed too rigidly Do skills teach procedures or inject missing facts?. Three approaches in the collection attack this. Creating skills inside the running agent loop grounds them in the exact task and reached 87.94% accuracy Does creating skills inside the agent loop eliminate mismatches?. A separately trained curator drifts toward execution logic and cross-task meta-strategies Can a separate trained curator improve skill libraries better than frozen agents?. That looks like an upper tier of the hierarchy forming on its own. SkillOpt edits skill documents like weights, keeping only changes that pass held-out validation, at zero added inference cost Can skill documents be optimized like neural network weights?.
Sources 10 notes
Resource2Skill distills tutorial videos, code repositories, and articles into a Skill Wiki that improved artifact quality by +11.9 percentage points across seven authoring domains compared to agents without skill access, outperforming two baseline approaches in most test cases.
Code2Skill extracts over 1 million auditable skills from GitHub repositories through source-blind reconstruction verification, improving performance from 42.90 to 47.90 macro-average across eight benchmarks, outperforming trajectory and document-derived alternatives.
MegaRAG builds hierarchical multimodal knowledge graphs from text and visuals to answer cross-chapter, global questions that flat chunk retrieval cannot reach. The hierarchy supports abstraction levels from high-level summaries to page-specific details while treating images as first-class graph nodes.
Versioned Capability Vectors embedded in HNSW indices couple semantic matching with policy and budget constraints, making capability discovery a first-class operation that scales sub-linearly as agent heterogeneity increases.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
Show all 10 sources
MetaGPT demonstrates that agents producing standardized engineering documents achieve superior coordination compared to conversational exchange. Active information pulling from shared environments eliminates noise and mirrors efficient human workplace infrastructure.
Analysis of 8,135 trials shows procedural anchoring accounts for 65.7% of skill cases versus 4.5% for knowledge injection. Skills fail when retrieved incorrectly, invoked out of context, or followed too rigidly.
MUSE-Autoskill demonstrates that invoking skill creation from within the agent's reasoning loop grounds new skills in exact task context, immediate feedback, and runtime validation. In-loop skills reach 87.94% task accuracy and transfer to other agents with minimal loss, eliminating the situated context problem of offline authoring.
SkillOS shows that separating a trainable curator from a frozen executor, grouped by task streams, causes skill repositories to shift from generic verbose additions toward actionable execution logic and cross-task meta-strategies. The trained curator generalizes across different executor backbones and domains.
SkillOpt treats skill documents as trainable external state of frozen agents, using a text-space optimizer with held-out validation gating to accept only edits that improve performance. Across 52 benchmark cells and seven models, the approach matches or exceeds baselines while adding zero inference cost and enabling transfer across models.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Demystifying Agent Skills: Why They Work-Until They Don't
- MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation
- RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources
- SkillOS: Learning Skill Curation for Self-Evolving Agents
- Grounded Skill Synthesis from Code at Scale for Agentic Intelligence
- SkillClaw: Let Skills Evolve Collectively with Agentic Evolver
- LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents
- Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose