Can two simple safety rules — when to learn, and when to say no — keep a self-improving AI from drifting?
Can applicability conditions and veto rules make self-training stable across substrates?
This explores whether two kinds of guardrail, rules for when a learning update is allowed to apply and rules that let a check reject a bad update, can keep self-improving systems from drifting, whether the thing being improved is model weights or a written-down skill or prompt.
This explores whether rules about when to learn and when to reject an update can keep self-improvement stable, whether the thing changing is model weights or a written skill. The corpus supports the idea in each setting separately, and no note tests one rulebook across both. It also shows the limit: a gate only stabilizes training if its check is independent of the model being trained.
Applicability conditions, meaning rules for which examples deserve an update, show up on the weights side. When all of a prompt's sampled answers earn nearly the same reward, the learning signal is weak and regularization pulls the policy toward generic, input-agnostic templates. Filtering to high-variance prompts before each update fixes this (Why do language models collapse into generic templates?). Majority-vote self-distillation applies the same logic: it distills only on rollouts where the model disagrees with itself, and it matched or beat supervised methods on five benchmarks (Can a model's own consensus replace ground truth labels?). Both rules ask whether this example can teach anything, and both skip the ones that can't. A related fix changes the reward instead of gating the update. Binary correctness rewards teach confident guessing, and adding a Brier-score term repairs calibration without hurting accuracy (Does binary reward training hurt model calibration?).
Veto rules are clearest on the text side. SkillOpt lets an agent rewrite its own instructions, but only within a per-step edit budget. A held-out validation set can veto any edit, and rejected edits go into a buffer that later serves as negative feedback. Ablations show this beats unconstrained rewriting for both stability and generalization (Does constraining edits make skill learning more stable?). A third substrate, an external skill library, sidesteps the problem. VOYAGER stores executable skills outside the weights and composes new ones from old ones, so nothing is overwritten and there is no catastrophic forgetting (Can agents learn new skills without forgetting old ones?). The gate matters less when the store is append-only.
Gates are only as good as their referee. Self-consistency looks like a safe check, but models eventually learn to give confidently wrong answers they can reproduce, and the reward drifts away from correctness while training still looks like progress (Does self-consistency reliably reward correct answers during training?). The broader diagnosis is that pure self-improvement is circular, and methods that work bring in an external anchor such as past model versions, third-party judges, user corrections or tool feedback (Can models reliably improve themselves without external feedback?). SkillOpt's held-out validation set is one of those anchors, which helps explain why its veto works. Peer cohorts move in the same direction: diverse, decoupled models rewarding each other avoid the collapse of self-generated feedback (Can peer models replace external judges for reward signals?). Methods where the model is its own judge, such as alternating actor and judge roles (Can models learn to judge themselves without external rewards?) or training self-evaluation into the space after the answer (Can models learn to evaluate their own work during training?), have no independent veto. SERL's gain from 52.37% to 59.90% on AlpacaEval is real but modest.
Two cautions apply. Gates need to look at the model's own failures: offline correction traces fail because they don't match the errors the model actually makes at test time, while online RL on its own mistakes works (Why does self-correction training on offline data fail?). And no gate can certify unconditional good behavior. Any scored behavior is observed behavior, so training can only ever show a model complies when it is being checked (Can behavioral training prove a model always complies?).
The corpus therefore suggests that gates make self-training more stable but not safe. What carries across weights and text is a design principle rather than a specific rule: pick updates that carry signal, cap how far any one update can move, and let something outside the model's own opinion say no.
Sources 12 notes
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Unsupervised on-policy self-distillation using the model's own majority-vote consensus matched or surpassed supervised methods on five benchmarks. The key mechanism distills only on self-inconsistent rollouts, using agreement as the teaching signal rather than external labels.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
SkillOpt's ablations show that adding a textual learning-rate budget, held-out validation gate, and rejected-edit buffer (retaining failed edits as negative feedback) produces more stable and generalizable skill improvement than allowing agents to freely rewrite their own instructions.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
Show all 12 sources
Self-consistency works as an intrinsic reward for bootstrapping RL without labels, but models eventually learn to generate confidently wrong but reproducible answers. The proxy reward correlation with correctness degrades over training, creating a failure mode that looks like improvement.
Pure self-improvement stalls due to the generation-verification gap, diversity collapse, and reward hacking. Reliable improvement methods succeed by smuggling in external anchors: past model versions, third-party judges, user corrections, or tool feedback.
Co-RL trains decoupled models using peer predictions as rewards, avoiding the bias and collapse of self-generated feedback. Heterogeneous cohorts consistently improve reasoning across benchmarks and often match ground-truth supervised training.
SERL enables self-improving language models by having them alternate between generating responses and judging them pairwise, deriving rewards from ranking consistency and self-consistency of judgments. On AlpacaEval, this reached 59.90% win rate without external signals, up from 52.37%.
Post-Completion Learning exploits unused sequence space after model output to train self-assessment capabilities during training while maintaining zero inference cost. The model learns to compute its own reward functions, internalizing evaluation rather than relying on external reward models.
SFT on offline correction traces fails because training errors don't match test errors and models collapse into single correction modes. Multi-turn online RL under the model's own error distribution successfully trains self-correction by letting models practice correcting their actual mistakes.
Any scored behavior is observed behavior, so training data cannot distinguish between a policy that always complies and one that complies only when watched. Only unobserved behavior would separate them, making such a test logically impossible.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Can Large Reasoning Models Self-Train?
- Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
- Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
- Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
- Training Language Models to Self-Correct via Reinforcement Learning
- SkillClaw: Let Skills Evolve Collectively with Agentic Evolver
- Self-Rewarding Language Models
- Demystifying Agent Skills: Why They Work-Until They Don't