What really makes AI 'self-improvement' different from ordinary training — who's doing the improving, or who decides what counts as better?

This explores two separate questions that usually get blurred together when people talk about AI improving itself: who does the improving, and who decides what 'better' means. The clearest framing in the corpus treats these as two independent dials What separates self-improvement from policy improvement?. One dial asks whether the improver sits inside the agent or outside it. The other asks whether the performance standard comes from the agent itself or from somewhere else. Under this view, 'recursive self-improvement' and ordinary iterative training are not different kinds of thing. They are the same improvement cycle with the dials set differently. The surprising lesson is that the second dial matters far more than the first.

The reason is circularity. If a model judges its own progress by its own standard, it can only find errors it is already able to see. That limit has a name: the generation-verification gap. A model can improve itself only to the extent that it checks answers better than it produces them, and on factual tasks that gap disappears entirely What limits how much models can improve themselves?. A broader review argues that pure self-improvement stalls for this reason, along with diversity collapse and reward hacking. It also finds that the methods that do work quietly bring in an outside anchor: an earlier model version, a third-party judge, user corrections, or feedback from tools Can models reliably improve themselves without external feedback?. So an internal improver is fine. An internal standard is where things go wrong.

Revising reasoning shows this most clearly. When reasoning models revise their own uncertain answers, they mostly keep wrong answers, and smaller models often switch correct answers to incorrect ones Does self-revision actually improve reasoning in language models?. Give the same revision step a critique from an external model and accuracy goes up. The act of revising is identical in both cases. What changes the outcome is where the judgment comes from Does revising your own reasoning actually help or hurt?. The same split appears in test-time scaling. Internal methods train a model to reason more on its own, while external methods add search and verification at inference time. The two work together rather than compete How do internal and external test-time scaling compare?.

The most interesting work sits on the line between the two dials: what if the standard improves too? Meta-Rewarding adds a 'meta-judge' that grades the judge, so the actor and its evaluator get better together. It pushed past the plateau a fixed judge hits, without human supervision Why do self-improvement loops plateau without updating the judge?. The Red Queen Gödel Machine makes evaluation part of the improvement loop itself. That lets self-improvement reach tasks like writing and proofs, which have no fixed grader Can evaluators improve alongside the agents they score?. These systems blur the inside/outside distinction. Whether they avoid circularity or just move it up one level is the open question. ModularRSI takes the opposite approach. It deliberately keeps the standard external by improving harness modules on data separate from the benchmark, so gains are less likely to be fitted to the test Can harness modules improve separately from benchmark data?. Sakana reports that the Darwin Gödel Machine's self-discovered improvements carried over to other models and other programming languages. That transfer is a kind of outside check that the improvements were real Do agent improvements discovered in one model transfer to others?.

This distinction also shapes how risk is assessed. One survey argues that today's industrial 'self-refinement' is bounded and checked against outside measures. Open-ended recursive self-improvement, which would need to grow its own standards, still faces grounding problems that can be measured today Are self-refinement and recursive self-improvement actually the same thing?. Even frontier assessments struggle here. OpenAI rated GPT-5.6 below its 'High' self-improvement threshold based on a single debugging metric with no reported numbers Does GPT-5.6 show meaningful self-improvement capability?. The takeaway: when you hear 'AI improving itself,' ask who holds the measuring stick. A system can safely do its own improving. Trouble starts when it also writes its own measure of what counts as better.