The safety math for self-improving AI assumes a fixed goal — what happens when the system gets to set its own?

This explores whether self-improvement systems still have their correctness guarantees when you remove the fixed, externally-defined goal and let the system evolve its own target instead. The corpus suggests the answer flips depending on what kind of guarantee you're asking about — and that the interesting move is not preserving the static target but replacing it with a moving one that's engineered to stay honest.

Start with why anyone would remove static utility in the first place. Fixed benchmarks and reward functions saturate: once an agent gets strong enough, a stationary target becomes something to game rather than achieve, and you get reward hacking Why do fixed benchmarks fail as agents get stronger?. So the static external utility isn't a safe anchor you'd want to keep — it's a liability that gets worse as the loop improves. The fix in that work (RQGM) is telling: it splits the search into epochs with a *stable within-epoch* criterion but *evolves* the utility across epoch boundaries. That's the key structural insight — you don't get a global guarantee, you get per-epoch guarantees, and you keep the target moving just ahead of the optimizer so it can never catch up and cheat.

But there's a deeper limit the question is circling. A whole cluster of the corpus argues that a self-improvement loop with *no* external signal at all is formally bounded no matter what — the generation-verification gap means a model can't reliably validate its own fixes, so metacognition alone hits a hard ceiling What stops large language models from improving themselves? What actually constrains large language models from self-improvement?. The methods that actually work don't remove external utility so much as smuggle it back in through side doors: past model versions, third-party judges, user corrections, or tool feedback Can models reliably improve themselves without external feedback?. So 'removing static external utility' and 'removing external grounding entirely' are two very different moves — the first is often healthy, the second breaks reliability by construction.

The most concrete answer to your literal question comes from the Darwin Gödel Machine, which *deliberately* trades away formal guarantees. The original Gödel machine required a formal proof that a self-modification improves utility before applying it; DGM abandons that proof requirement and substitutes empirical benchmarking plus an evolutionary archive of variants Can AI systems improve themselves through trial and error?. It got 2.5× on SWE-bench — so yes, removing the static, provable utility does break the *formal* guarantee, and in this case that was the point. What replaces it is an empirical, statistical form of confidence, which buys open-endedness at the cost of certainty.

The surprising takeaway: the formal guarantee was never really doing the useful work. The systems that improve best keep two things the theory says you need — a *stable criterion inside each step* and an *external validator* — while throwing away the one thing that felt safest, the fixed global objective. And there's a wrinkle worth knowing: the ability to keep evolving the loop isn't uniform across models. Weak models can't even invoke their own scaffolding, strong models over-follow instructions, and the capacity to actually benefit from self-updates peaks in the middle Do stronger models always evolve harnesses better?. So whether removing static utility helps or hurts also depends on which model is running the loop.