You can change what an AI does by rewriting its instructions in minutes, or by retraining it over weeks — how do they differ?
How do fast skill injection and slow gradient updates work on different timescales?
This explores two ways of changing what an AI system can do, and how they fit together. One is editing the text and tooling around a frozen model (fast, at deployment time). The other is updating its weights with gradients (slow, in training).
This explores two ways of changing what an AI system can do: editing the skills, prompts and tooling around a frozen model (fast, at deployment time), and updating its weights with gradients (slow, in training). The corpus has no single note that sets the two clocks side by side, but the pieces line up. On the fast side, Can execution harnesses lift model performance without retuning weights? shows that optimizing the execution system around frozen weights lifts several models on Terminal-Bench 2.1, and the same runbook carries over to newer models unchanged. The limit is spelled out in How do knowledge injection methods trade off flexibility and cost?. Prompt-level approaches need no training but only activate what the model already knows. Baking knowledge into weights is fastest to run but costly and inflexible, and retrieval-style injection is the flexible middle at the price of latency.
The slow side is smaller and more stubborn than you might expect. Does reinforcement learning update only a small fraction of parameters? finds that RL touches only 5 to 30 percent of parameters, yet the same sparse, nearly full-rank subnetworks show up across random seeds. Slow learning is selective in a repeatable way, not diffuse. It also locks things in. Does RL training collapse format diversity in pretrained models? shows RL amplifying one pretraining format within the first epoch while collapsing the alternatives. Does RL training follow a predictable two-phase learning sequence? shows it moving from procedural mastery to strategic planning as its own internal schedule. A weight update is a commitment carried into every future prompt. A skill edit is a note you can tear up tomorrow.
The most interesting move is the fast layer borrowing the slow layer's discipline. Does constraining edits make skill learning more stable? gives skill rewriting a textual learning-rate budget, a held-out validation gate, and a buffer of rejected edits kept as negative feedback. That is gradient-training hygiene applied to prose, and it beats letting agents rewrite their own instructions freely. Can a separate trained curator improve skill libraries better than frozen agents? nests the clocks explicitly. A curator is trained slowly with RL, the executor stays frozen, and the skill library is edited fast. The curator learns to move the library from generic, verbose additions toward actionable execution logic and cross-task meta-strategies, and it transfers across different executor models.
Some methods sit between the two clocks. Can models dynamically activate expert skills at inference time? trains small expert vectors slowly by tuning singular values, then mixes them on the fly at inference without interference, beating LoRA with fewer parameters. The fast part is choosing among slowly learned pieces. Fast injection also depends on finding the right skill in the first place. What blocks skill retrieval in task decomposition? finds that standard LLM decomposition reaches only 34% step-level recall, and that gates whether retrieval works at all.
Speed doesn't buy safety. Does reward hacking always stem from the same failure? finds the same failure in weight training, output selection and prompt revision: optimizing against a signal that only partly captures the real task. A fast skill loop can be gamed just as a slow RL loop can. Its validation gate is only as trustworthy as the score it checks against.
Sources 10 notes
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
Dynamic injection (RAG) maximizes flexibility but adds latency; static embedding is fastest but costly and inflexible; modular adapters balance efficiency with swappability; prompt optimization requires no training but only activates existing knowledge. Combining all three outperforms any single approach.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Show all 10 sources
SkillOpt's ablations show that adding a textual learning-rate budget, held-out validation gate, and rejected-edit buffer (retaining failed edits as negative feedback) produces more stable and generalizable skill improvement than allowing agents to freely rewrite their own instructions.
SkillOS shows that separating a trainable curator from a frozen executor, grouped by task streams, causes skill repositories to shift from generic verbose additions toward actionable execution logic and cross-task meta-strategies. The trained curator generalizes across different executor backbones and domains.
Transformer2 demonstrates that tuning only singular values within weight matrices produces composable expert vectors that dynamically mix at inference without interference, outperforming LoRA with fewer parameters and enabling continual specialization.
Standard LLM decomposition reaches only 34% step-level recall, gating retrieval success. Correcting step count recovers 75% of gains in iterative methods, shifting the bottleneck to representation-level reranking rather than vocabulary alignment.
Reward hacking arises during weight training, output selection, and prompt revision through a shared failure: optimization against signals that incompletely represent the actual task. The substrate matters less than the misalignment between the scoring function and ground truth.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Art of Scaling Reinforcement Learning Compute for LLMs
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- SkillClaw: Let Skills Evolve Collectively with Agentic Evolver
- Reinforcement Learning with Rubric Anchors
- Demystifying Agent Skills: Why They Work-Until They Don't
- Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose
- Understanding Reasoning from Pretraining to Post-Training
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs