Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement

Abstract. When we speak of recursive self-improvement (RSI), are we speaking of a phenomenon, a mechanism, or a prospect? Towards autonomous and evolving intelligence, RSI is being claimed at many scales, while no single framework that formally describes these emerging instances exists. Its counterpart in the classical realm, iterative policy improvement, is characterized by generalized policy iteration (GPI), a framework of broad applicability with well-understood theoretical properties, but only where the update principle and the evaluation base lie outside the agent. In this paper, we propose Generalized Agent Iteration (GAI), a formal framework that describes iterative policy improvement and RSI as two cases of a single learning paradigm. GAI defines the agent as a configuration of modifiable components within a system and models the learning process as a cycle of agent evaluation and agent improvement. Two pivotal dials then distinguish the instances: whether the improving mechanism is part of the agent and whether the standard it is measured against is grounded outside it. The former dial delineates the boundary between GPI and RSI, and the latter determines a system’s polarity as anchored, goal drift, or fully self-referential.

Introduction. Self-improvement is an old yet persistent ambition in artificial intelligence. It is the premise of Good’s ultraintelligent machine, the last invention that human beings would need to make (Good, 1965), and the engine of the later accounts in which a system that improves its own improvement mechanism compounds into a rapid rise in capability (Yudkowsky, 2013). How such systems are built, and where they fail, is therefore worth stating precisely. Nowadays, self-improvement is claimed at many scales: agents that refine their own outputs (Madaan et al., 2023; Shinn et al., 2023); systems that rewrite the routine that improves them, from code-level self-modification (Robeyns et al., 2025; Zelikman et al., 2023) to self-referential agent frameworks (Kakade et al., 2026; Yin et al., 2024; Zhang et al., 2025, 2026); research loops that search programs or experiments under fixed evaluators (Lu et al., 2024; Novikov et al., 2025); and, at the far end, systems that co-evolve the standard they are judged by (Iacob et al., 2026) or propose to do without one (Schaul, 2024).

Discussion / Conclusion. Scope and Limitations The framework describes how a system changes, not how to compute the change. The configuration space of Section 3.1 is a reference idealization: writing that an agent instance is drawn from the modifier states what is chosen, not that the choice is reachable or affordable. Four limits follow. First, the definitions carry no reachability or complexity claims, and the defects above are properties of the formulation; where an existing system exhibits one, we cite it rather than measure it. Second, we do not model time scales: the alternation between world actions and agent updates is counted in steps, not compute. Third, practical guards, such as the proof gate of the Gödel machine or a coding benchmark, are treated as external mechanisms that restrict what a modifier may propose; whether a guard should itself be an agent component is a design choice we leave open.