Demystifying Agent Skills: Why They Work-Until They Don't

Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: When do skills help, why do they work, and where do they fail? Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and crossframework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7% of skill cases, versus 4.5% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts.

Introduction. Large language model (LLM) agents are increasingly expected to improve through experience rather than solve each task from scratch. Recent agent systems therefore store and reuse traces of prior execution: environment setup sequences, tooluse patterns, debugging routines, and verification steps that were discovered in earlier runs. This shift is especially appealing for tool-using agents (Yao et al., 2023; Schick et al., 2023; Qin et al., 2024; Wang et al., 2024a; Yang et al., 2024), where repeated failures often arise not from a lack of highlevel reasoning, but from rediscovering the same procedural details again and again. Among the proposed memory forms, skills(Anthropic, 2025) have emerged as a particularly compelling abstraction. A skill is not simply a record of past execution, but a compact description of what to do, what to check, and what pitfalls to avoid. Compared to storing raw execution traces or direct workflow memories, skills promise three advantages.

Discussion / Conclusion. This paper studies the behavior of skills through controlled experiments and contrastive trajectory analysis, moving beyond aggregate success rates to ask when skills help, why they work, and where they fail. Our results show that skills are most effective as procedural anchors and can also fail when they are retrieved incorrectly, invoked in the wrong context, followed too rigidly, or used on tasks that require deeper reformulation and runtime validation. Overall, our findings suggest that skill use should be understood as a lifecycle problem rather than a single memory-injection mechanism. Building better self-evolving agents requires not only generating more skills, but also improving how agents represent, retrieve, and leverage procedural knowledge. We hope this analysis provides a foundation for more principled evaluation and design of future skill-based agent systems.