Simple checks like 'does it compile' keep a self-improving AI from spinning in circles — but do they also keep it safe?

This explores whether a cheap outside check, like 'does the code compile' or 'did the test score go up', is enough to keep a self-improving AI on track and safe. The corpus answers in two parts. Weak anchors are clearly necessary: without them, self-improvement goes in circles. They are clearly not enough for safety, and the reason is not what most people expect.

Start with why any anchor matters. Pure self-improvement stalls. A model grading its own work runs into a gap between what it can generate and what it can verify, its outputs lose variety, and it learns to game its own reward. Every method that does work sneaks in an outside signal: an older model version, a separate judge, user corrections, or tool feedback Can models reliably improve themselves without external feedback?. Simple checks do a lot of work here. The Darwin Gödel Machine dropped formal proofs and kept only empirical benchmarking, and it still more than doubled its coding performance Can AI systems improve themselves through trial and error?. SICA climbed from 17% to 53% on SWE-Bench by editing only its own scaffold, the code and prompts wrapped around the model Can a single agent improve itself by editing its own code?. STOP adds a subtler point: the *form* of the checking signal, source code versus plain English, changes how much improvement the loop can extract Can language models improve their own scaffolding without weight updates?. So for capability, weak anchors carry you a long way.

Safety is a different problem, and three things stand out. First, a compile check judges each change on its own. It can't see a problem that only shows up across a sequence of changes. Per-action guardrails cannot even *express* constraints that depend on history; only monitors that track behavior over time can Can stateless checks ever catch sequence-level constraint violations?. A thousand edits that each compile can still add up to a system nobody signed off on. Second, in SICA the agent edits its own *oversight code*. When the anchor sits inside the thing being improved, it is no longer fully outside. Third, the push to internalize evaluation, for example training models to compute their own rewards Can models learn to evaluate their own work during training?, moves the anchor further inward. That saves cost, but it gives up some of the independence that made the anchor useful. And passing a check is not the same as being competent: reasoning models that reflect fluently still score only 20-23% on problems that need real backtracking Can reasoning models actually sustain long-chain reflection?.

The more promising answer in the corpus is to keep anchors weak but add structure around them. SkillOpt limits how much an agent may change per step (a kind of learning-rate budget for text), gates each edit on held-out validation, and keeps rejected edits as negative examples. That works better than letting the agent rewrite itself freely Does constraining edits make skill learning more stable?. Splitting a vague goal into checkable sub-criteria also makes weak signals harder to game Can breaking down instructions into checklists improve AI reward signals?. Reversibility matters too. Most recent progress happens in the 'fast loop' of prompts, memory, and tools rather than in model weights, partly because those changes can be undone Do self-improving agents really split into two distinct loops?. This also explains why the field separates bounded, testable self-refinement (what industry does today) from open-ended recursive self-improvement Are self-refinement and recursive self-improvement actually the same thing?. Weak anchors are enough for the first and not the second. That second, open-ended case is where Anthropic has called for slowing down Does recursive self-improvement pose serious risks to society?.

The takeaway you may not have expected: the danger with compile checks is not that they are too weak to drive progress. They drive progress well. The danger is that they check each step and never the path, and in systems that edit themselves, the checker itself can be edited. The corpus has no direct study that measures how often weak-anchor loops drift into unsafe behavior, so that part remains an argument from structure rather than measured evidence.