If an AI can only tweak its own instructions in small steps, does a bad first draft trap it?
Does a tight learning-rate bound prevent skills from escaping poor starting points?
This explores whether capping how much an agent can rewrite its own skill instructions per step (a 'textual learning rate') keeps it stuck near a weak first draft, or whether small controlled steps can still climb out.
This explores whether capping how much an agent can rewrite its own skill instructions per step (a 'textual learning rate') keeps it stuck near a weak first draft. The corpus doesn't test this directly, so the honest answer is that nothing here shows a tight bound trapping a skill, and the nearest evidence points the other way. SkillOpt's ablations found that a textual learning-rate budget, a held-out validation gate and a rejected-edit buffer together gave more stable and more generalizable skill improvement than letting agents rewrite instructions freely Does constraining edits make skill learning more stable?. It doesn't report what happens when the starting skill is very poor, so 'the bound doesn't trap it' is an inference, not a finding.
The rejected-edit buffer is the part that matters for escaping. Failed edits aren't thrown away. They're kept as negative feedback, so even a rejected small step tells the agent where not to go next. Two other notes show the same idea from different angles. SkillRL keeps successes as concrete demonstrations and turns failures into abstracted lessons Should successful and failed episodes be processed differently?. LEAP deliberately errs on few-shot examples and derives explicit principles from the mistakes Does learning from mistakes improve in-context learning?. In each case, progress under small steps comes from making failures informative, not from taking bigger steps.
In weight-space RL, the risk that shows up is lost adaptability and weak signal, not tightness. Models kept close to their base distribution (up to 70% closer) retained the plasticity to learn later tasks, while parameter-only approaches that drifted further stalled when the domain changed Does staying close to the base model preserve learning ability?. Getting stuck can also come from a flat learning signal. When reward variance within a prompt is low, the task gradient weakens and regularization takes over, so the policy settles into generic templates. The fix was picking higher-variance prompts, not loosening the step size Why do language models collapse into generic templates?.
Big steps have their own cost. Overly hard RLVR samples let rare accidental successes look like high-advantage trajectories, which reinforces shortcuts that then contaminate capabilities the model already had Do overly hard RLVR samples actually harm model capabilities?. RL also tends to lock onto one dominant pretraining format within the first epoch, and the winner tracks model scale rather than performance Does RL training collapse format diversity in pretrained models?. Early moves can therefore fix a bad starting point in place, which is an argument for gating steps. The corpus doesn't say how tight a budget can get before it slows escape from a start that is truly far from good. The other lever it does show is richer exploration data, since training on failed attempts and backtracking teaches more robust reasoning than shortcut solutions alone Can models learn better by training on messy exploration paths?.
Sources 8 notes
SkillOpt's ablations show that adding a textual learning-rate budget, held-out validation gate, and rejected-edit buffer (retaining failed edits as negative feedback) produces more stable and generalizable skill improvement than allowing agents to freely rewrite their own instructions.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
LEAP demonstrates that models achieve better performance on reasoning and math tasks by intentionally erring on few-shot examples, reflecting on mistakes, and deriving explicit task-specific principles—without additional labeled data or fine-tuning.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Show all 8 sources
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Research shows that training on messy trajectories—failed attempts, self-correction, and backtracking—teaches more robust reasoning than training only on shortcut solutions. This approach models o1-style deep reasoning as search internalization rather than solution memorization.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- Self-distillation Enables Continual Learning
- In-Context Principle Learning from Mistakes