If an AI keeps rewriting its own instructions, one bad edit can snowball — so how do you keep it in check?

This explores how limiting how much, and how, an agent can rewrite its own skill documents keeps self-improvement from compounding its own mistakes. The corpus's most direct answer is SkillOpt, which treats a skill document like a model's weights and trains it with three guardrails. A textual learning-rate budget caps how much can change per step. A held-out validation gate accepts an edit only if it improves performance on tasks it wasn't tuned on. A rejected-edit buffer keeps failed edits around as negative feedback. In the ablations, this bounded setup gave more stable and more generalizable improvement than letting the agent freely rewrite its own instructions Does constraining edits make skill learning more stable? Can skill documents be optimized like neural network weights?.

The word "cascade" is my framing, not the notes'. But the corpus describes the pieces of one. In self-editing, every rewrite becomes the starting point for the next, so one bad edit gets built on instead of staying isolated. The notes name three ways this goes wrong. Recursive edits to agent scaffolding can memorize the training tasks, so gains shrink out of distribution Does harness self-improvement memorize tasks instead of learning broadly?. Reward hacking shows up whether you update weights, select outputs, or revise prompts, because the score being optimized only partly represents the real task Does reward hacking always stem from the same failure?. And pure self-improvement is circular: it stalls on the generation-verification gap and diversity collapse Can models reliably improve themselves without external feedback?.

Each control maps onto one of those failures. The budget limits how far a single bad edit can move the document. The held-out gate is the outside check the self-improvement literature says every reliable method has to smuggle in. Because it's held out, it also blocks the memorization and score-gaming failures, since the agent can't tune the skill to the exact tasks that grade it. The gate leans on the fact that checking an edit is easier than writing a good one, which is the same gap that limits self-improvement What limits how much models can improve themselves?. That gap vanishes for purely factual tasks, so a gate like this helps most where you can actually measure whether a skill got better. The rejected-edit buffer adds a different protection. Failed edits become information, so the optimizer doesn't keep re-proposing the same bad idea.

There's a reason to be careful with skill text in particular. Skills mostly work as procedural anchors that stabilize what an agent does, rather than injecting missing facts, and they fail when retrieved wrongly or followed too rigidly Do skills teach procedures or inject missing facts?. My inference is that a corrupted skill then misdirects every task that retrieves it, which is why small, validated changes matter. A related design separates the editor from the worker. A trained curator maintains the skill library while the executor stays frozen, and over training the curator moves toward actionable execution logic and cross-task strategies instead of verbose additions Can a separate trained curator improve skill libraries better than frozen agents?.

The bigger picture is that these controls are what keep self-improvement in the safe category. A large survey separates bounded, evaluable self-refinement, which is what industry does today, from open-ended recursive self-improvement, which runs into grounding, collapse, and compute limits Are self-refinement and recursive self-improvement actually the same thing?. Capping edits, gating them on held-out results, and remembering rejections is a practical recipe for staying in the first category. The notes don't say whether it scales beyond it.