If an AI keeps rewriting its own work with no outside check, why does it slowly drift off course?

This explores why an AI that keeps rewriting its own outputs or instructions, with nothing outside checking each change, tends to wander, either by memorizing quirks of recent cases (overfitting) or by piling up conflicting changes (incoherence). The corpus has one study that tests the full loop and several that explain the ingredients. In short: the editor is also the judge, each edit is built on the leftovers of the last one, and nothing separates an edit that helps in general from one that just fixes the latest complaint.

Start with the judge problem. Models over-trust answers they generated themselves, because high-probability outputs feel more correct when the model evaluates them, so self-review tends to rubber-stamp Why do models trust their own generated answers?. It gets worse with repetition. When a model reconsiders its own earlier reasoning, it grows more confident in its errors rather than less. Debate among genuinely different models reverses this, improving both accuracy and calibration Does a model improve by arguing with itself?. So a self-edit loop isn't a fresh look each round. It's the model agreeing with itself, and confidence drifts upward while quality doesn't.

Then the errors compound. When a model's own past mistakes fill its context, later error rates climb non-linearly. Scaling the model doesn't fix this, and only extra test-time thinking dampens it Do models fail worse when their own errors fill the context?. Iterative refinement has the same structure as overthinking: each pass adds noise with no guarantee of improvement. Compressing memory between passes beat simply running longer at matched compute Do iterative refinement methods suffer from overthinking?. An unchecked edit loop is exactly this, because each revision's flaws become the raw material for the next. Edits also don't cleanly overwrite what's already there. Synthetic training documents add new associations predictably, but revising existing ones is unpredictable Can training data edits reliably override what models already believe?. That is a plausible route to incoherence, with new patches layered on old beliefs instead of replacing them. The corpus doesn't test that for self-edit loops directly, so treat it as a reading, not a finding.

The most direct evidence is on what stops the drift. SkillOpt's ablations compare agents freely rewriting their own instructions against agents given three guards. The guards are a textual learning-rate budget, a held-out validation gate, and a buffer of rejected edits kept as negative feedback. The guarded version improved more stably and generalized better Does constraining edits make skill learning more stable?. Each guard plausibly answers one failure mode. The budget caps how far any single step can move things. The gate is the outside judge, since an edit has to help on cases it wasn't written from, which is what catches overfitting. The buffer keeps the loop from retrying ideas it already tried and dropped.

Two other notes point at what the guards are protecting. Self-correction trained on offline correction traces fails because the training errors don't match the model's real ones, and it collapses into a single correction habit. It works when the model practices on its own actual mistakes with online reinforcement learning Why does self-correction training on offline data fail?. And self-generated signal isn't poison in itself. Consistency training uses the model's own clean-prompt responses as targets, and it works because those responses are a fixed reference the model must match Can models learn to ignore irrelevant prompt changes?. What an unchecked self-edit loop lacks is that anchor: an external test, a held-out check, or a fixed reference to pull each edit back toward reality.